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CHAPTER  I.  INTRODUCTION 


Artificial  intelligence  (AI)  is  one  of  the  emerging  technologies  of  the 
19S0s.  AI  is  the  use  of  computers  to  perform  tasks  typically  considered  to  require 
intelligence.  This  form  of  computing  concerns  the  manipulation  of  symbols  which 
encode  the  knowledge  and  the  competence  to  perform  intelligent  acts,  rather  than 
the  familiar  form  of  computing  that  concerns  numerical  calculation  and  record 
keeping. 


Almost  all  computing  done  today  involves  numerical  calculation;  for 
example,  work  in  the  physical  sciences  and  engineering.  However,  a  vast  amount 
of  human  productivity  involves  symbol  manipulation:  solving  legal  problems, 
making  medical  diagnoses,  designing  systems,  or  planning  genetic  engineering 
experiments.  Edward  Feigenbaum  has  said  (Swaine,  1983,  p.  12),  "Symbolic 
manipulation  will  become  the  dominant  type  of  computer  use  simply  because 
there  are  so  many  more  things  to  be  done  compared  with  mere  calculation."  When 
will  this  happen?  Feigenbaum  predicts,  "if  not  by  1990,  perhaps  by  the  turn  of  the 
century."  Think  of  the  absolutely  vast  amount  of  computation  that  goes  on  today, 
and  then  imagine  that  amount  of  computing,  or  more,  being  devoted  to  symbol 
manipulation.  This  is  the  future  of  AL 

At  this  level  of  generality,  the  requirements  of  the  Air  Force  and  of  the 
other  Armed  Services  are  no  different  from  society  at  large:  much  work  that 
needs  to  be  done  is  symbolic  in  nature.  This  report  focuses  on  computer 
applications  to  the  inherently  symbolic  (as  contrast  to  computational)  tasks  of 
designing,  developing  and  delivering  instruction;  monitoring  and  assessing 
personnel  performance;  and  assisting  personnel  in  the  performance  of  their  jobs. 
That  is,  the  operations  and  maintenance  of  Air  Force  systems  provide  both 
challenges  and  opportunities  for  the  application  of  AI  technology. 


Background  of  This  Report 


This  paper  is  the  first  step  in  a  9-month  program  planning  effort,  tasked 
to  develop  a  research,  development,  and  applications  program  for  the  utilization 
of  artificial  intelligence  technology  in  training,  performance  assessment  and  job 
performance  aiding.  The  objectives  of  this  paper  are  (a)  to  introduce  the  reader 
to  AI  and  those  subfields  of  AI  that  are  most  relevant  to  this  research  and 
development  (R&D)  program,  (b)  to  report  on  relevant  ongoing  R&D  sponsored  by 
the  Department  of  Defense  (DoD),  (c)  to  establish  the  challenges  facing  the  Air 
Force  in  the  three  target  areas  and  explain  how  each  presents  opportunities  for  AI 
applications,  (d)  to  draw  out  important  practical  concerns  which  must  be  faced  by 
an  AI  R&D  program,  and  finally,  (e)  to  draw  this  analysis  together  by  proposing  a 
set  of  recommendations  for  building  an  AI  applications  program  in  training, 
performance  assessment  and  job  performance  aiding. 


The  brief  history  of  AI  spans  about  25  years.  Early  projections  of 
competent  AI  applications  in  language  translation  and  chess  were 
disappointments;  AI  proved  to  be  much  harder  than  expected.  Understandabiyt  a 
backlash  of  skepticism  was  generated  by  these  early  failures.  But,  AI  profited 
from  experience.  Its  research  thrust  was  redirected.  It  was  recognized  that 
computational  brute  force  and  generality  are  not  the  keys  to  solving  important 
problems.  The  beginning  of  APs  success  story  was  the  recognition  that  knowledge 
powers  the  solution  to  important  problems. 

By  the  late  1970s,  this  redirection  brought  success  to  AI  research 
endeavors.  AI  programs  were  developed  that  performed  successfully  or 
demonstrated  promise  in  a  number  of  different  applications  areas;  for  example, 
medical  diagnosis,  speech  understanding,  spectroscopy,  and  symbolic 
mathematics.  By  1980,  the  potential  of  AI  was  recognized,  and  active  AI  research 
programs  were  underway  in  several  large  corporations.  In  1981,  Japan  announced 
a  major  commitment  to  AI  applications  with  its  fifth  generation  computer 
project.  The  first  AI  applications  business  ventures  were  launched.  By  1982,  AI 
could  be  considered  to  be  entering  a  period  of  industrialization  and  public 
awareness.  Numerous  AI  business  ventures  had  been  launched  and  the  promise  of 
AI  had  been  touted  in  at  least  six  major  broad  circulation  periodicals. 

Thus,  AI's  time  has  come.  An  examination  of  potential  AI  applications  in 
the  DoD  in  general,  and  in  training,  performance  assessment,  and  job  performance 
aiding  in  particular,  is  timely  and  justifiable. 


Artificial  Intelligence 


Definition 


AI  is  the  study  and  application  of  what  is  known  about  intelligence  to  the 
development  of  computer  systems  that  model  intelligent  behavior.  Intelligent 
behavior  involves  tasks  for  which  algorithmic  solutions  do  not  exist.  Such  tasks 
often  involve  complexity  (designing  a  bridge),  uncertainty  (deciding  whether  to 
buy  or  sell  on  today's  stock  market)  or  ambiguity  (filling  in  the  details  while 
reading  a  novel).  Usually,  a  controlled  search,  rather  than  a  random  or  exhaustive 
search,  for  a  solution  is  necessary.  Great  amounts  of  knowledge  are  typically 
required  for  successful  performance.  AI  is  concerned  with  several  types  of 
knowledge,  principally  procedural  knowledge  (how-to-do-it),  declarative 
knowledge  (knowledge  of  function  or  structure),  and  control  knowledge  (general 
problem  solving  strategies). 

Research  in  AI  is  being  conducted  in  two  disciplines  with  complementary 
objectives:  computer  science  and  cognitive  science.  As  a  branch  of  computer 
science,  the  study  of  AI  seeks  to  develop  new  tools  and  new  techniques  that  will 
support  the  use  of  computers  to  do  tasks  that  require  intelligence.  As  a  major 
branch  of  the  emerging  discipline  of  cognitive  science,  the  study  of  AI  serves  to 
provide  controlled  environments  for  experimenting  with  methods  and  tools  used  in 
studying  human  intelligent  behavior. 
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The  major  source  of  knowledge  (about  intelligence)  is  the 
experimental  study  of  human  behavior  to  discover 
mechanisms*  followed  by  construction  of  simulation  systems 
(e.g.,  computer  programs)  to  verify  whether  the  discovered 
mechanisms  do  indeed  have  the  properties  inferred.  (Newell* 

1973,  p.  14.) 

As  humans,  we  are  our  own  best  exemplars  of  intelligent  behavior.  By 
following  our  example*  we  are  learning  how  to  teach  machines  to  help  us  do  our 
work. 


A1  Elements 

Representing  knowledge  and  techniques  for  processing  knowledge  are  the 
two  basic  elements  of  AL 

Depicting,  symbolizing,  structuring  and  communicating  knowledge  have 
clearly  become  a  focus  in  the  short  history  of  A I: 

The  fundamental  problem  of  understanding  intelligence  is  not 
the  identification  of  a  few  powerful  techniques,  but  rather 
the  question  of  how  to  represent  large  amounts  of  knowledge 
in  a  fashion  that  permits  their  effective  use  and  interaction. 

(Goldstein  &  Papert,  1977,  p.  85) 

The  second  element,  processes  that  operate  on  representations*  is 
predicated  on  the  principle  that  intelligent  behavior  is  objective-directed  or  goal¬ 
seeking  (e.g.,  in  problem  solving  ...  a  solution;  in  speech  understanding.  .  . 
meaning;  in  medical  consultation  ...  a  correct  diagnosis  and  prescription).  The 
attempt  to  reach  a  goal  requires  first  creating,  then  searching  an  "area"  or  space 
within  which  the  goal  will  lie.  Creating  the  space  requires  the  building  of  a 
representation  of  the  problem  or  situation.  In  chess  this  representation  could  be  a 
tree  of  all  possible  legal  moves  and  countermoves.  After  creating  a 
representation,  a  search  of  the  space  is  conducted.  Controlling  search  is  a  major 
AI  issue. 


Traditional  AI  Applications 

While  all  work  in  AI  is  ultimately  founded  on  the  basic  elements  of 
representation  and  process,  the  field  has  a  number  of  traditional  applications 
areas.  Robotics  is  a  basic  applications  area,  involving  planning  and  executing 
physical  motions.  Vision  is  another,  involving  encoding  visual  scenes  into 
meaningful  and  useful  representations.  Another  applications  area  involves 
language,  both  understanding  and  generating  written  and  spoken  natural  language 
(that  is,  English,  French,  etc.).  Understanding  spoken  language  is  an  extremely 
difficult  problem  that  has  served  as  an  excellent  test  bed  for  AI  ideas,  but  the 
problem  is  far  from  solved. 


The  development  of  systems  that  learn  from  experience,  can  reason  by 
analogy,  and  can  recognize  and  manage  their  own  resources  and  limitations  are 
also  important  areas  of  AI  research.  The  development  of  AI  tools  is  naturally  a 
field  of  intense  interest.  Software  tools  include  programming  languages, 
environments,  and  special  purpose,  high  level  systems.  Related  to  software  are  AI 
efforts  to  generate  computer  programs  automatically  from  English  language-like 
specifications.  Hardware  is  also  a  tool.  In  the  last  few  years,  special  purpose 
LISP  machines  have  become  commercially  available  on  which  to  do  AI 
development  work.  Further  work  on  parallel  processing  hardware  systems  is  an  AI 
research  priority. 

One  AI  applications  area  that  just  recently  gained  wide  recognition  is 
expert  systems.  The  growth  of  AI  RicD  in  the  next  few  years  will  almost 
certainly  be  dominated  by  this  applications  field.  An  expert  system  is  an  AI 
program  that  exhibits  or  aids  professional  level  human  expertise  in  such  areas  as 
medical  diagnosis,  structural  engineering,  petroleum  exploration,  or  computer 
system  configuration.  The  development  of  expert  systems  has  become  a 
commercial  enterprise,  and  to  the  extent  that  these  systems  do  aid  humans  in  the 
work  they  must  do,  expert  systems  will  proliferate. 

Expert  systems  have  become  successful  outside  the  research  laboratory, 
prior  to  other  AI  applications  fields,  because  these  systems  work  in  very  narrowly 
constrained  environments.  For  example,  the  vocabulary  and  nature  of  discourse  in 
medical  diagnosis  are  tightly  circumscribed  in  comparison  to  the  vocabulary  and 
discourse  that  appear  in  the  newspaper  every  day.  The  vocabularies  associated 
with  professional  activities  provide  a  way  to  begin  modeling  the  expertise 
involved.  But  with  vision  or  speech  understanding,  the  knowledge  of  how  these 
things  are  done  is  much  less  developed. 

Developing  expert  systems,  an  activity  known  as  "knowledge 
engineering,"  is,  even  so,  not  an  easy  task.  For  most  genuine  expertise,  even 
though  there  is  a  vocabulary  to  describe  what  is  done,  there  is  no  way  to  state 
how  performance  is  achieved.  People  know  more  than  they  are  aware  of  knowing. 
Because  the  details  of  expertise  are  beyond  current  understanding,  knowledge 
engineering  is  as  much  the  psychological  science  of  unraveling  the  mysteries  of 
intelligent  human  performance  as  it  is  the  computing  science  of  creating  software 
exhibiting  it. 

Summary.  What  AI  field  is  most  appropriate  to  applications  in  training, 
performance  assessment  or  job  aiding?  The  answer  lies  in  the  nature  of  the 
problem.  The  subject  under  discussion  is  technical  expertise,  and  the  AI  field  of 
expert  systems  has  the  most  to  offer.  Expert  systems  are  new,  and  in  many  ways 
fragile.  Much  work  needs  to  be  done  on  increasing  their  robustness.  But,  these 
systems  have  demonstrated  their  worth.  This  is  not  yet  the  case  for  natural 
language  interfaces  or  vision  systems.  Vision  and  language  interfaces  will 
enhance  the  utility  of  expert  systems,  but  it  is  the  representation  and  utilization 
of  technical  knowledge  that  shall  always  remain  at  the  core  of  what  AI  has  to 
offer  the  operations  and  maintenance  of  Air  Force  systems. 


AI  Literature 


The  Handbook  of  Artificial  Intelligence  (Barr  4c  Feigenbaum,  1982)  is  a 
basic  source  of  information  on  AI.  The  handbook  "is  intended  to  be  an 
introductory  reference  book  both  for  researchers  within  AI  and  for  those  outside 
who  need  a  guide  to  the  unchartered  waters  of  AI  research  and  literature"  (Price, 
1982,  p.  2).  It  is  not  a  textbook,  collection  of  separately  authored  papers,  or  a 
synthesis  of  the  field.  The  handbook  itself  has  a  section  about  the  AI  literature, 
which  is  recapitulated  here. 

The  AI  literature  is  generally  categorized  chronologically:  early 
materials  -  before  1965,  middle  materials  -  from  1965-1975,  recent  materials  - 
1975  to  present. 

Early  publications.  Books  published  during  the  early  period  include 
Computers  and  Thought  (Feigenbaum  4c  Feldman,  1963)  and  Semantic  Information 
Processing  (Minsky,  1968).  A  classic  book  of  this  period  is  Newell  and  Simon's 
(1972)  Human  Problem  Solving,  which  has  been  heralded  as  the  beginning  of  the 
discipline  of  cognitive  science,  the  synthesis  of  the  fields  of  artificial  intelligence 
and  cognitive  psychology. 

Middle  publications.  Books  of  the  middle  period  (1966-1975)  include 
Understanding  Natural  Language  (Winograd,  1972),  Computer  Models  of  Thought 
and  Language  (Schank  4c  Colby,  1973),  The  Psychology  of  Computer  Vision 
(Winston,  1975),  and  Representation  and  Understanding  (Bobrow  4c  Collins,  1975). 
A  general  introduction  to  the  field  of  AI  during  this  period  includes  Sciences  of 
the  Artificial  (Simon,  1969). 


Recent  publications.  General  A I  introductions  during  this  period  include 
Godel,  Escher,  Bach:  An  Eternal  Golden  Braid  (Hofstadter,  1979)  and  books  by 
Norman  and  Rumelhart  (1975)  and  Schank  and  Abelson  (1977),  which  develop  from 
first  principles  systems  for  natural  language  understanding.  Textbooks  about  AI 
include  Winston's  (1977)  Artificial  Intelligence,  and  Nilsson's  (1980)  Principles  of 


Artificial  Intelligence.  Books  dealing  with  LISP  programming  include  Charniak, 
Riesbeck,  and  McDermott's  (1980)  Artifical  Intelligence  Programming,  and  Schank 


Five  Programs  Plus 


Cognitive  Process,  and  The  Psvcholo 


Moran  4c  Newell 


of  Human-Computer  Interaction  (Card 


Very  recently  a  number  of  excellent  reference  sources  have  been 
published  concerning  expert  systems.  Two  books  on  this  subject  are  Knowledge- 
Based  Systems  in  Artificial  Intelligence  (Davis  4c  Lenat,  1983),  and  Building 
Expert  Systems  (Hayes-Roth,  1983).  Additionally,  there  are  the  following 
technical  reports  and  journal  articles:  'The  Organization  of  Expert  Systems,  A 
Tutorial"  (Stefik  et  al.,  1982),  "An  Investigation  of  Tools  for  Building  Expert 
Systems"  (Waterman  4c  Hayes-Roth,  1982),  and  "An  Overview  of  Expert  Systems" 
(Gevarter,  1982). 


Journals,  periodicals,  and  conference  proceedings.  The  most  current 
Information  about  work  in  AI  is  found  in  journals,  periodicals,  and  conference 
proceedings.  These  journals  and  periodicals  include  Artificial  Intelligence;  the 
journal  of  the  Cognitive  Science  Society,  Cognitive  Science;  the  American  Journal 
of  Computational  Linguistics;  the  SIGART  Newsletter,  published  by  the 
Association  for  Computing  Machinery;  the  American  Association  for  Artificial 
Intelligence  (AAAI)  AI  Magazine;  the  International  Journal  of  Man-Machine 
Studies  (IJMMS)  which  also  publishes  a  great  number  of  papers  directly  relevant  to 
the  application  of  AI  to  instruction;  and  several  publications  of  the  Institute  of 
Electronic  and  Electrical  Engineers  (IEEE). 

Associations,  conferences,  and  proceedings  include  the  biennial 
International  Joint  Conference  on  Artificial  Intelligence  (IJCAI),  and  the  national 
meetings  of  the  AAAI.  The  Cognitive  Science  Society  holds  annual  meetings  as 
well.  Carnegie  Mellon  University  annually  holds  a  meeting  on  cognitive  science, 
and  the  National  Institute  of  Education  and  the  U.S.  Department  of  Education 
periodically  sponsor  seminars  addressing  issues  relevant  to  the  application  of  A!  to 
instruction. 

Bibliographies.  The  bibliographies  of  the  three  volumes  of  The  Handbook 
of  Artificial  Intelligence  referred  to  at  the  beginning  of  this  chapter  provide  a 
comprehensive  key  to  the  bulk  of  published  literature  on  AL  Each  topical  section 
of  the  handbook  includes  a  section  indicating  important  references  pertinent  to 
that  section.  Of  particular  interest  to  the  study  of  applications  of  AI  to 
instruction  is  the  handbook  section  on  applications-oriented  AI  research  in 
education. 


CHAPTER  II.  CURRENT  RELEVANT 
DoD  RESEARCH  AND  DEVELOPMENT 


A  number  of  Department  of  Defense  agencies  are  becoming  involved  in 
R&D  relevant  to  the  AI  applications  under  consideration  in  this  report.  This  R&D 
is  summarized  in  Table  1,  which  lists  the  sponsoring  agency,  project  title,  and 
performing  organization  if  different  from  sponsoring  agency  and  known.  In 
Appendix  A  the  detail  of  this  list  is  expanded  to  include  the  names  and  telephone 
numbers  of  responsible  individuals  (of  both  sponsoring  agency  and  performing 
organization),  project  durations,  levels  of  funding,  and  pertinent  sources  of 
additional  information.  Narrative  descriptions  of  relevant  projects  are  integrated 
into  the  main  body  of  this  report. 

The  R&D  reported  in  Table  1  is  not  strictly  limited  to  projects  explicitly 
identified  with  artificial  intelligence.  Also  reported  is  R«ScD  in  the  three  areas  of 
interest  (training,  performance  measurement,  and  job  performance  aiding)  which 
forms  a  technology  base  for  applications.  The  research  reported  also  spans  the 
continuum  from  basic  theoretical  work  in  cognitive  psychology  to  projects  that 
have  near-term  or  midterm  applications. 


DoD  Centers  of  Activity 


As  can  be  seen  from  Table  1,  DoD  is  sponsoring  substantial  R&D  in  the 
target  Al  applications  fields.  The  intent  of  this  section  is  to  summarize,  at  the 
program  level,  what  each  of  the  involved  agencies  is  doing  and  is  planning  to  do. 


Office  of  Naval  Research  (ONR) 

A  major  center  of  relevant  activity  is  the  Office  of  Naval  Research 
(ONR),  Psychological  Sciences  Division,  Personnel  and  Training  Research 
Program.  This  program  includes  areas  in  theory-based  personnel  assessment, 
information  processing  abilities,  instructional  theory  and  advanced  training 
systems,  and  cognitive  processing.  R&D  in  each  of  these  areas  is  directly 
relevant  to  the  target  AI  applications  areas.  In  particular,  "work  on  instructional 
theories  looks  to  eventual  application  in  generative  knowledge-based,  automated 
systems  for  training. . (Office  of  Naval  Research,  1981,  p.  1). 

Additionally,  ONR  and  the  Army  Research  Institute  are  jointly  funding  a 
program  in  intelligent  computer-based  instruction  (ICAO.  This  3-year  program, 
running  from  FY  82  to  FY  85,  has  a  total  program  funding  level  of  about  $2.5 
million. 


With  respect  to  the  future,  program  planning  at  ONR  is  looking  foward 
to  the  funding  of  a  center  of  excellence  for  ICAI.  The  goal  of  this  center  would 
be  the  development  of  A I  tools  for  the  creation  and  delivery  of  all  types  of 
instructional  material,  both  print  and  interactive. 
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ONR  is  planning  to  start  applied  work  in  intelligent  maintenance  training 
simulation  and  intelligent  job  aids.  This  program  will  focus  on  the  principles  of 
human  machine  intelligence  for  computer-based  training  and  job  aiding.  Finally, 
an  effort  is  contemplated  to  combine  R4cD  in  several  funding  categories  in  order 
to  bring  together  a  coordinated  effort  bearing  on  the  following  areas:  (l)  adult 
basic  skills,  (2)  technical  training  with  an  emphasis  on  computer-assisted 
instruction  and  intelligent  maintenance  simulations,  (3)  problem  solving  aspects  of 
tactical  training,  and  (4)  the  development  of  instructional  materials,  both  print 
and  interactive. 


Navy  Personnel  Research  and  Development  Center  (NPRDC) 

NPRDC  has  three  program  areas  specifically  relevant  to  this  report: 
personnel  and  occupational  measurement,  instructional  technology,  and  training 
systems.  Relevant  projects  underway  at  NPRDC  are  listed  in  Table  1.  A  project 
with  the  great  direct  relevance  to  this  report  is  "Project  STEAMER—Advanced 
Computer-Based  Training  for  Propulsion  and  Problem  Solving.1' 

The  objective  of  this  project  is  to  develop  an  advanced  computer-based 
instruction  system  for  a  steam  propulsion  plant,  including  a  detailed  math  model 
of  a  propulsion  system,  a  graphics  interface,  and  teaching  strategies.  The 
complete  STEAMER  math  model  with  graphics  interface  has  been  implemented  on 
development  LISP  machines.  Development  of  a  microcomputer-based  STEAMER 
training  system  is  underway.  STEAMER  is  a  long-term,  high  investment  project 
which  has  potential  for  adding  considerably  to  the  current  state  of  the  art  of 
intelligent  computer-assisted  instruction. 

Another  project  worth  noting  is  the  Authoring  Instructional  Materials 
(AIM)  project.  This  has  great  commonality  with  the  FY  85  ONR  thrust  area  in  the 
development  of  instructional  materials.  It  is  expected  that  these  two  efforts  will 
be  tightly  coordinated. 


Navy  Training  Equipment  Center  (NTEC) 

AI  applications  in  training  are  being  pursued  at  NTEC.  Here,  emphasis  is 
being  placed  on  tackling  some  of  the  problems  associated  with  team  training. 
Specifically,  the  concept  of  Al-based  "surrogates"  is  being  developed.  For 
example,  consider  a  surrogate  opponent.  In  one-on-one  submarine  warfare 
training  simulations,  a  simulated  knowledgeable  opponent  could  enhance  the 
effectiveness  of  training  by  freeing  the  instructors  from  running  the  offensive 
submarine  to  perform  other  instructionai  roles. 

The  surrogate  team  member  concept  is  also  being  explored.  When  all 
members  of  a  team  are  novices,  the  learning  rate  for  the  team  as  a  whole  can  be 
slow.  Having  an  individual  trainee  join  a  "surrogate  team”  (which  already  has 
competence)  can  accelerate  the  trainee's  rate  of  skill  acquisition. 


In  addition  to  the  surrogate  concept,  AI  applications  at  NTEC  are 
focused  on  performance  assessment.  "Assessment"  should  be  interpreted  as  going 


beyond  the  mere  collection  of  data  to  the  analysis  and  use  of  performance  data  in 
providing  instructional  direction  and  feedback  to  trainees. 

AI  applications  in  job  performance  aiding  will  most  naturally  integrate 
with  ongoing  advances  in  JPA  technology.  Of  note  is  the  Personal  Electronic  Aid 
for  Maintenance  (PEAM)  and  other  miniaturized,  interactive  job  aiding  devices 
under  investigation  at  NTEC. 

As  at  ONR  and  NPRDC,  work  focusing  on  the  instructional  design  and 
development  process  is  also  being  pursued  at  NTEC,  in  particular,  the  computer- 
based  system,  CASDAT  (Computer  Aided  System  for  Development  of  Aircrew 
Training). 


Training  Analysis  and  Evaluation  Group  (TAEG) 

Work  in  the  area  of  computer-aided  authoring  of  instructional  materials 
is  also  underway  at  the  TAEG,  which  is  located  on  the  same  site  as  NTEC.  Work 
is  targeted  on  automated  development  strategies  for  the  following  kinds  of  tasks: 
procedural,  paired  associate,  and  visual  object  classification. 

TAEG  is  also  involved  with  the  Naval  Technical  Information  Presentation 
Program,  a  5-year,  multimillion  dollar  program  to  improve  the  Navy's  efficiency 
in  publishing.  This  effort  is  managed  by  the  David  W.  Taylor  Naval  Ship  Research 
and  Development  Center  and  is  now  near  completion. 


Army  Research  Institute  (ARI) 

The  joint  ICAI  projects  with  ONR  have  already  been  mentioned.  ARI  has 
funded  the  Adaptive  Computerized  Training  System  (ACTS),  designed  to  train 
electronics  troubleshooting  procedures.  Beyond  this,  ARI,  as  of  this  writing,  has 
no  funded  projects  in  AI  relevant  to  the  target  areas.  However,  in  preparation  is 
a -project  termed  AI-Based  Maintenance  Tutor  (AIMT)  which  is  to  function  as  tutor 
and  job  aid. 


Air  Force  Human  Resources  Laboratory  (AFHRL) 

The  AFHRL  R&D  program  is  divided  among  four  thrust  areas:  (a) 
staffing  and  force  management,  (b)  air  combat  tactics  and  training,  (c) 
maintenance  and  combat  support,  and  (d)  training  design  and  delivery.  AI 
applications  in  training,  performance  measurement,  and  job  aiding  are  potentially 
relevant  to  each  of  these  thrust  areas.  For  example,  if  the  scope  of  "performance 
measurement”  is  expanded  to  include  personnel  actions  to  be  taken  based  on  such 
measurements,  much  of  R&D  going  on  in  the  manpower  and  force  management 
area  contributes  a  technology  base  for  potential  AI  applications.  In  this  R&D 
area,  sample  projects  include  task-oriented  measurement  technologies;  evaluation 
of  individual  performance  in  mechanical  specialties;  and  the  development  of  a 
prototype,  computer-based  training  decisions  system. 
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Ongoing  research  in  air  combat  tactics  and  training  also  provides 
opportunity  for  potential  AI  applications,  in  particular,  a  desk-top  trainer 
demonstration  for  utilization  in  pilot  training  and  microcomputer-based  special 
function  trainer.  A  project  focused  on  automated  performance  measurement  for 
the  C-5  aircraft  offers  the  basis  for  extension  of  "measurement  systems"  to 
"assessment  systems,"  where  data  are  not  only  collected,  but  are  analyzed  and 
used  to  provide  instructional  feedback.  There  is  apparent  potential  for 
collaborative  efforts  with  NTEC  in  this  area.  As  indicated  in  the  introduction  to 
this  paper,  AFHRL  is  planning  a  program  of  AI  research,  development,  and 
application. 

The  "surrogate"  concept  is  also  applicable  to  team  training  for  operators 
of  command,  control,  and  communications  systems.  Another  project  that  builds 
toward  potential  AI  applications  concerns  the  development  of  draft  military 
specifications  for  maintenance  task  analysis  and  logic  tree  troubleshooting  aids. 
If  this  guidance  document  were  implemented  as  an  intelligent,  interactive 
development  system,  many  potential  advantages  to  AI  applications  would  result. 
A  project  with  similar  relevance  concerns  the  development  of  computer-based 
maintenance  aids  for  technicians.  Much  work  in  the  field  of  maintenance 
simulation  affords  opportunity  for  AI  applications  because  such  simulations  can 
help  shift  the  focus  of  simulation  from  the  equipment  to  the  task,  where 
automated  instructional  intervention  can  be  provided  in  response  to  student 
performance  data.  Project  data  for  the  efforts  mentioned  above  are  presented  in 
both  Table  1  and  in  Appendix  A. 

As  indicated  in  the  introduction  to  this  paper,  AFHRL  is  planning  a 
program  of  AI  research,  development  and  application.  This  paper  addresses  AI 
opportunities  in  training,  performance  measurement  and  job  performance  aiding. 
A  parallel  program  planning  effort  is  focusing  on  applications  of  AI  in  the  areas  of 
maintenance  diagnostics,  intelligent  maintenance  information  systems  and  in 
built-in  test  and  automatic  test  equipment.  These  two  parallel  efforts  are  being 
coordinated  because  it  is  realized  that  engineering  design,  maintenance,  logistics, 
and  training  all  are  closely  related  and  interacting  subcomponents  of  effective 
and  ready  weapons  systems. 


Inter-Agency  Coordination 

AI  is  new,  and  applications  efforts  even  newer.  Therefore,  formal 
mechanisms  for  inter-agency  coordination  are  not  yet  fully  developed  and  further 
work  on  coordination  is  needed.  Two  obvious  justifications  of  such  coordination 
are  (a)  the  problems  faced  by  the  three  armed  services  in  training,  performance 
assessment  and  job  performance  aiding  are  similar  and  amenable  to  similar 
solutions,  and  (b)  the  resource  pool  of  trained  people  to  do  A I  applications  work  is 
severely  limited. 

A  good  start  toward  coordinating  AI  applications  in  training  and 
maintenance  is  the  Working  Group  in  Simulation,  Training  and  Maintenance  of  the 
Panel  on  Technology  Initiatives  in  Artificial  Intelligence,  one  of  seven  panels  of 
the  Joint  Directors  of  Laboratories  (JDL)  Panel,  a  subgroup  of  the  Joint  Logistics 


Commanders.  The  charter  of  the  JDL  is  to  put  together  joint  service  projects. 
Each  of  its  seven  panels  has  four  working  groups:  electronic  warfare;  robotics; 
command,  control,  and  communications;  and  simulation  and  training.  Therefore, 
there  is  in  existence  an  interservice  coordination  committee  that  is  directly 
relevant  to  the  coordination  of  AI  applications  in  the  target  areas.  The  current 
working  group  consists  of  representatives  from  ARI,  NTEC,  NPRDC,  and  AFHRL. 


CHAPTER  III.  CHALLENGES  AND  APPROACHES 


Introduction 


The  three  training  issues  of  concern  to  this  study,  training,  performance 
assessment,  and  job  performance  aids  (JPAs)  are  actually  integrated  pieces  of  the 
total  Air  Force  training  model.  In  the  simplest  description  of  the  relationship 
among  the  three  elements,  training  provides  the  basis  for  baseline  job 
performance  expertise,  which  is  in  turn  augmented  and  developed  through  on-the- 
job  training  (OJT).  The  effectiveness  of  either  form  of  Air  Force  training  is 
assessed  through  some  form  of  performance  measurement.  The  JPAs,  notably 
technical  orders,  support  the  knowledge  and  skills  of  personnel  by  providing 
procedural  aids  for  operations  and  maintenance  tasks.  Numerous  trade-offs  are 
required  among  (a)  which  knowledge  and  skills  should  be  taught  formally  in 
residence  training,  (b)  which  should  be  acquired  through  OJT  and  experience,  and 
(c)  which  need  not  be  specifically  taught  because  TOs  and  other  JPAs  are 
available.  It  is  necessary  to  look  at  the  entire  picture  both  instructional^  and 
institutionally  to  arrive  at  a  reasonable  set  of  compromises  concerning  trade-offs. 

A  common  denominator  among  the  three  areas  of  interest  is  task 
analysis:  it  is  the  basis  for  training,  evaluation,  and  JPAs.  Therefore,  task 
analysis  is  singled  out  as  a  separate  issue  area  in  which  to  consider  the  potential 
of  AI  applications. 

Since  training,  measurement,  and  job  performance  are  embedded  in  a 
larger  organizational  matrix,  a  systems  analysis  is  appropriate  in  analyzing  the 
challenges  and  opportunities  in  these  areas.  Two  of  the  most  important 
considerations  from  the  systems  point-of-view  are  the  issues  of  validation  and 
system  maintenance. 

Validation  is  centrally  important  in  training,  performance  measurement, 
and  JPAs.  Often  the  most  effective  validation  paradigms  involve  crossing  the 
boundaries  among  these  three  fields,  as  when  a  training  program  is  evaluated  in 
terms  of  the  on-the-job  performance  of  its  trainees. 

An  issue  related  to  validation  is  that  of  maintenance.  For  example,  as 
field  evaluations  show  training  to  be  deficient  in  certain  areas,  the  training 
programs  must  be  maintained  to  accommodate  these  deficiencies.  If  a  program  of 
applications  of  AI  technologies  were  to  focus  solely  on  training,  individual 
assessment,  and  JPAs,  then  important  and  consequential  opportunities  for  the 
application  of  AI  technology  would  be  overlooked;  the  ultimate  payoff  of 
applications  in  any  of  the  three  basic  areas  might  ultimately  be  discounted  by 
forces  operating  on  the  larger  systems  level. 

This  section  is  organized  around  challenges  and  opportunities  related  to 
task  analysis,  training,  individual  assessment,  job  performance  aiding,  and  the 
integration  of  these  areas  into  one  system.  Within  each  of  these  areas,  a  brief 
description  of  relevant  challenges  is  developed,  followed  by  a  discussion  of 


potential  approaches  to  the  application  of  AI  in  meeting  these  challenges. 
Related  past,  current,  and  planned  DoD  R&D  in  each  area  is  integrated  into  the 
discussion  of  approaches. 


Methodology 

Information  regarding  the  challenges  facing  the  Air  Force  training, 
assessment,  and  job  performance  support  communities  has  been  gathered  from 
various  sources.  These  sources  include  informed  briefings  from  AFHRL  personnel, 
input  from  consultants  (Dr.  Peter  G.  Poison,  Department  of  Psychology, 
University  of  Colorado;  Dr.  Mark  L.  Miller,  Vice  President,  Computer* Thought 
Corporation,  Dallas,  Texas),  the  technical  literature,  and  discussions  with  DoD 
and  civilian  professionals  working  in  these  areas. 

Sources  of  information  regarding  approaches  to  meeting  the  challenges 
identified  include:  the  past,  current,  and  planned  research  activities  in  the 
application  of  AI  to  these  areas.  Other  sources  are  advice  from  consultants 
regarding  the  more  and  less  feasible  approaches  to  specific  problems  and  a 
general  synthesis  of  the  potential  for  application  of  AI  technology  to  the 
challenges  identified. 


Task  Analysis 


Task  analysis  is  the  common  denominator  of  training,  individual 
assessment,  and  job  performance  aiding.  As  embodied  in  the  Tri-Service 
Instructional  Systems  Development  (ISD)  approach,  task  analysis,  in  the  form  of  a 
set  of  objectives,  is  the  basis  for  the  development  of  both  instructional  materials 
and  criterion  referenced  tests  of  mastery.  Task  analysis  also  plays  an  important 
role  in  performance  assessment,  as  well  as  in  the  development  of  job  performance 
aids. 


Naturally,  the  exact  definitions  of  task  analysis,  as  well  as  the 
methodologies  and  procedures  for  carrying  them  out,  vary  from  training,  to 
assessment,  to  job  performance  aiding  environments.  For  example,  within  the 
training  environment,  an  analysis  of  the  prerequisites  to  new  objectives  and 
desirable  sequencing  of  learning  events  may  be  considered  a  part  of  task  analysis 
(Gagne,  1977).  However,  in  the  fields  of  performance  measurement  and  JPAs, 
prerequisites  and  sequencing  are  less  relevant. 

Setting  such  differences  aside,  common  to  training,  assessment,  and 
performance  support  is  the  need  to  identify  the  specific  behavioral  and 
information  processing  actions  a  human  must  accomplish  in  effective  and 
efficient  performance  of  a  task  or  job.  Because  of  this  commonality,  application 
of  AI  technology  to  the  challenges  of  task  analysis  will  benefit  all  three  areas 
individually. 


Challenges 

What  are  the  challenges  facing  task  analysis?  Successful  solutions  to  the 
various  training  and  maintenance  problems  in  the  Air  Force  require  a  correct 
description  of  the  kinds  of  knowledge  that  are  necessary  to  carry  out  the  various 
tasks  and  the  development  of  a  training  program  that  in  fact  imparts  this 
knowledge.  Informal  analyses  of  these  training  requirements,  while  insightful  and 
often  correct,  lack  the  rigor  and  the  detail  necessary  to  develop  training  programs 
that  are  independent  of  the  expertise  of  a  given  individual.  Informal  analyses 
(even  when  conducted  by  very  competent  individuals)  often  turn  out  to  be 
glaringly  incomplete  because  gaps  (which  must  be  accomplished  by  the  novice)  are 
filled  in  implicitly  by  the  set  of  experts  who  prepare  the  training  material.  They 
understand  the  task,  they  write  a  description  that  seems  reasonable  to  them,  but 
since  they  already  know  how  to  accomplish  the  task,  they  have  no  way  of 
evaluating  themselves  to  see  if  all  the  steps  to  accomplishing  the  task  are 
explicit. 


As  a  remedy  for  this,  an  adequate  validation  can  be  carried  out  as  a  part 
of  the  ISD  cycle,  but  this  feedback  loop  is  slow  because  the  effects  of  the  training 
courses  must  be  observed  on-the-job  and  reported  back  to  Air  Training  Command 
(ATC)  before  adjustments  to  the  training  can  be  considered. 

Another  task  analysis  problem  that  impacts  both  training  and  assessment 
is  the  difficulty  of  developing  a  task  analysis  that  is  generalizable.  For  example, 
a  survey  of  B-52  refueling  procedures  identified  seven  different  techniques  at 
seven  different  bases—all  technically  ccrrect.  What  is  common  among  these 
techniques?  What  motivates  the  discrimination  among  them?  What  should  be 
taught,  given  that  teaching  all  seven  different  possibilities  is  infeasible?  What 
standard  should  be  employed  in  assessing  performance  of  this  task? 

Task  analysis  is  critically  relevant  in  the  area  of  performance 
assessment.  A  major  problem  in  this  area  is  identifying  what  is  to  be  measured. 
Often  what  is  measured  is  based  on  an  analysis  of  the  job  by  one  isolated  content 
expert.  According  to  some  observers,  this  approach  has  failed  for  the  Army  in  its 
Skill  Qualification  Test  program.  A  critical  question  is  how  to  get  a 
representative  decomposition  of  a  task.  How  many  experts'  performances  must  be 
analyzed  before  a  sufficiently  broad-base  description  of  performance  is  in  hand? 
In  the  author's  opinion,  the  Air  Force  Occupational  Survey  Data  do  not  break  down 
tasks  into  enough  detail  to  be  a  solution  to  this  problem. 

The  problem  of  task  analysis  is  not  confined  to  differences  in  task 
approaches  within  one  task  area.  There  is  a  need  to  identify  subtasks  common 
among  sets  of  tasks,  so  that  training  efficiency  can  be  maximized  through  a 
proper  clustering  and  sequencing  of  training  objectives.  The  Air  Force  needs  to 
have  an  accurate  model  of  the  expertise  in  its  technical  corps  on  which  to  base 
manning  decisions.  The  03T  performance  records  must  be  more  descriptive, 
recording  more  than  limited,  simple  numeric  scores.  A  tracking  system  is  needed 
that  can  support  realistic  and  effective  deployment  of  technical  expertise.  Such  a 
system  must  be  based  on  detailed  task  analyses. 
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Some  tasks  are  easier  to  analyze  than  others.  At  one  end  of  a  continuum 
are  jobs  or  tasks  that  are  completely  routine,  fixed-sequence  procedures, 
incorporating  known  actions  and  methods  in  the  service  of  predetermined,  specific 
goals.  Normal  disassembly  of  a  piece  of  equipment  would  fall  in  this  category;  so 
would  using  a  word  processor  to  edit  a  manuscript.  At  the  other  extreme  are 
tasks  for  which  it  is  difficult  or  impossible  to  specify  the  goal,  component  actions, 
and  methods.  Many  management  jobs  fall  at  this  extreme  of  the  continuum;  so  do 
many  troubleshooting  and  problem  solving  tasks  such  as  fault  isolation  in  digital 
equipment  or  writing  and  debugging  computer  programs.  Let  us  call  those 
simpler,  easier  tasks,  routine  tasks,  and  the  complex,  difficult  to  define  tasks, 
genuine  problem  solving  tasks.  Approaches  to  the  challenges  of  task  analy-!* 
differ,  depending  on  whether  the  tasks  are  routine  or  are  genuine  problem  solving. 


Routine  tasks.  Kieras  and  Poison  (1982a,  1982b)  have  developed  a 
theoretical  approach  to  the  formal  analysis  of  task  complexity  in  routine 
cognitive  tasks.  The  tasks  of  particular  interest  to  their  project  deal  with 
interactive  computing.  Their  approach  is  built  on  and  extends  previous  work  in 
cognitive  psychology,  cognitive  science,  and  A I.  It  is  a  relevant  synthesis  and 
application  of  prior  work  in  these  fields  to  task  analysis. 


The  approach  involves  the  development  of  a  formal  description  of  the 
user's  knowledge  of  how  to  use  an  interactive  device  and  a  formal  description 
system  for  representing  the  interactive  device  itself.  The  user's  knowledge  of  the 
task  and  the  device  are  represented  using  formal  representation  schemes  from  A I. 
Details  of  both  of  these  representations  are  presented  in  Kieras  and  Poison 
(1982b).  The  formal  descriptions  are  computer  simulation  programs.  These  two 
descriptions  are  validated  by  running  one  against  the  other.  In  other  words,  the 
formal  description  of  the  user's  knowledge  (e.g.,  the  task  representation)  is 
validated  by  seeing  if  it  can  interact,  in  the  service  of  some  useful  goal  (e.g., 
editing  a  text),  with  the  formal  description  of  the  device.  If  either  description  is 
incomplete  or  inaccurate,  the  two  simulations  will  not  properly  mesh;  that  is,  they 
will  not  run,  or  they  will  produce  erroneous  results.  This  approach  is  appropriate 
only  for  routine  cognitive  skills— those  for  which  it  is  possible  to  completely 
specify  a  running  simulation  of  the  task.  Routine  cognitive  tasks  are  certainly 
required  in  many  career  specialties  for  which  this  approach,  without  modification 
or  extension  is  applicable.  Even  when  jobs  do  contain  genuine  problem  solving 
components,  the  routine  tasks  can  be  singled  out  and  treated  with  the  Kieras  and 
Poison  formalism. 


Given  the  two  models  of  the  Kieras  and  Poison  formalism,  a  number  of 
metrics  of  task  difficulty  can  be  derived.  One  way  that  task  difficulty  can  be 
interpreted  is  by  the  size  of  the  how-to-do-it  knowledge  representation.  Another 
way  is  to  count  the  depth  of  the  number  of  subgoals  that  need  to  be  invoked  to 
accomplish  that  task.  Once  these  measures  of  task  difficulty  are  known, 
instructional  resources  can  be  allocated  in  proportion  to  the  difficulty  of  the 
tasks.  Moreover,  the  task  representation  for  any  subtask  gives  a  precise 
specification  of  exactly  what  must  be  learned  in  order  to  perform  that  task.  Such 
metrics  not  only  help  in  deciding  where  to  place  instructional  emphasis,  they  also 


20 


help  in  deciding  exactly  what  it  is  that  must  be  taught.  In  fact,  it  may  be  fairly 
straightforward  to  translate  the  task  representation  into  instructional  text  or 
other  media. 

The  metrics  are  of  similar  use  in  evaluation.  Through  exploration  with 
simulations,  it  would  be  possible  to  develop  a  criterion  task  that  exercises  an 
arbitrary  proportion  of  the  task  hierarchy.  Hence,  efficient  and  effective 
performance  measurement  tests  can  be  designed.  Also,  since  technicians  may 
often  be  the  least  competent  on  those  tasks  which  require  the  most  learning,  such 
tasks  can  be  placed  under  special  scrutiny,  especially  if  they  are  critical  to  the 
job. 

Another  outcome  from  the  Kieras  and  Poison  approach  is  the  knowledge 
that  follows  from  a  mapping  of  the  task  representation  to  the  device 
representation.  If  there  is  a  mismatch  between  these  two,  there  probably  will  be 
difficulty  in  learning  the  correct  procedures  in  the  areas  of  the  mismatch.  This 
can  be  corrected  by  either  redefining  the  way  the  task  is  represented  (re-doing 
the  instruction),  or  by  redesigning  the  device  itself.  In  this  manner,  the  man- 
machine  interface  can  be  optimized  before  a  device  is  developed.  In  a  similar 
vein,  since  task  complexity  is  "induced"  by  the  design  and  structure  of  the  device 
(as  represented  by  the  device  representation),  prior  to  prototype  development  of  a 
device  its  representation  can  be  manipulated,  observing  the  effect  on  the  induced 
task  complexity.  The  goal  of  such  an  exercise  would  be  to  settle  on  a  device 
design  which  produced  the  minimum  human  operator  task  complexity. 

In  addition  to  the  task  and  device  representations,  an  important 
representation  is  of  the  user's  understanding  of  "how-it-works"  knowledge  for  a 
device.  The  knowledge  contained  in  a  mental  model  of  a  device  is  a  source  of 
meaningful  explanations  of  an  individual's  procedural  knowledge,  the  task 
representation.  Such  mental  models  help  novices'  initial  attempts  to  learn  how  to 
operate  a  device.  The  how-it-works  representation  is  a  hierarchy  of  explanations. 
Kieras  and  Poison  (1982a)  stipulate  several  criteria  for  selecting  those 
explanations  from  the  how-it-works  hierarchy  that  should  be  selected  for 
presentation  to  a  novice. 

In  sum,  the  Kieras  and  Poison  formalism  provides  a  method  for 
developing  and  validating  task  analyses  of  routine  cognitive  tasks  that  can  be  used 
(a)  to  direct  the  design  of  new  devices,  (b)  to  identify  subtasks  that  will  be 
difficult  to  learn,  (c)  to  specify  what  must  be  learned,  and  (d)  to  specify  what 
how-it-works  knowledge  would  be  useful  explanation  to  novices. 

Genuine  problem  solving  tasks.  For  the  Air  Force  the  need  to  support 
training,  measurement,  and  job  performance  is  more  critical  for  genuine  problem 
solving  than  for  routine  tasks.  In  briefings  by  staff  members  at  the  AFHRL— 
Training  Systems  Division,  avionics  maintenance  in  particular  was  mentioned  as  in 
need  of  support.  In  spite  of  advances  in  automated  test  equipment,  this  job  often 
requires  genuine  problem  solving  competency.  Avionics  maintenance  is 
representative  of  a  larger  class  of  genuine  problem  solving  tasks,  namely,  fault 
isolation,  diagnosis,  and  troubleshooting  tasks.  There  is  a  large  amount  of  R&D 
experience  with  training  and  performance  in  these  areas  that  could  be  pulled 


together  to  form  the  foundation  of  AI  applications  effort  in  training,  assessment, 
and  job  performance  aiding  in  these  critical  task  areas. 

The  task  analysis  of  genuine  problem  solving  tasks  can  be  better 
understood  by  considering  the  work  that  has  been  accomplished  by  researchers  in 
cognitive  psychology  and  AI.  Large  strides  in  the  theoretical  understanding  of 
problem  solving  behavior  have  resulted  from  the  synergistic  relationship  between 
these  two  fields.  Work  in  this  area  has  moved  away  from  the  domain  of  games 
(checkers,  the  Tower  of  Hanoi,  chess,  etc.)  and  has  focused  on  genuine  problems. 
Much  work  by  psychologists  has  focused  on  problem  solving  in  domains  such  as 
algebra  or  physics.  Work  in  areas  of  "real  world"  practical  significance,  combined 
with  an  emphasis  more  on  performance  than  on  theoretical  development,  results 
in  the  familiar  "expert  system." 

The  methodologies  and  techniques  used  by  both  psychologists  and 
"knowledge  engineers"  are  of  use  to  the  Air  Force  in  developing  approaches  to 
formal  analysis  of  genuine  problem  solving  tasks.  The  one  pre-eminent 
consideration  is  that  in  the  knowledge  about  a  certain  field  or  domain  lies  most  of 
the  power  to  solve  problems  in  that  domain. 

Psychologists  have  developed  a  modern  cognitive  theory  of  problem 
solving  in  the  tradition  of  information  processing  psychology  (Chi  &  Glaser,  1979; 
Greeno,  1976a,  1976b,  1977;  Hinsley,  Hayes  &  Simon,  1976;  Larkin,  1979a,  1979b, 
1980a;  Larkin,  McDermott,  Simon  Sc  Simon,  1979a,  1979b;  Mayer,  1974,  1975, 
1980;  Mayer  Sc  Greeno,  1972;  Mayer,  Greeno  Sc  Stiehl,  1975;  Newell  Sc  Simon, 
1972).  Problem  solving,  in  this  tradition,  is  characterized  as  understanding  and 
search  (Poison  Sc  Jeffries,  1980).  Expert  "understanding"  in  physics  problem 
solving,  for  example,  involves  the  use  of  a  qualitative  scientific  representation  of 
the  phenomena  at  hand  as  a  guide  to  the  selection  of  a  method  of  solution  (Chi, 
Feltovich  Sc  Glaser,  1980;  Greeno,  1980;  Larkin,  1980b;  Richardson,  1981). 
"Search,"  for  the  expert,  has  the  character  of  building  toward  a  goal,  rather  than 
working  backwards  from  a  goal. 

A  problem  solver  (expert  or  novice)  when  faced  with  a  problem,  has  a 
representation  in  memory  to  which  he  or  she  fits  the  problem.  Depending  on  the 
degree  of  meaningfulness  of  that  fit,  the  solver  then  brings  to  bear  a  number  of 
search  processes  that  act  on  the  representation,  filling  out  a  path  to  a  solution. 
Depending  upon  the  degree  of  meaningfulness  inherent  in  the  representation  to 
which  the  solver  fits  the  problem,  the  search  may  be  straightforward  or  it  may  be 
circuitous;  a  solution  may  result  forthwith  or  it  may  elude  the  solver  entirely. 
Often  a  problem  representation  must  be  adjusted  in  the  course  of  developing  a 
solution,  for  example,  from  a  functional  representation  to  a  physical 
representation,  or  from  a  conceptual  scientific  one  to  a  mathematical  one. 

It  is  important  to  note  that  the  representation  (also  termed  cognitive 
structure  or  schema)  brought  to  bear  on  the  problem  is  an  organized  collection  of 
experience  with  this  and  similar  problems.  Thus,  problem  solving  is  knowledge- 
dependent  or  knowledge-based.  Skill  in  problem  solving  is  not  reducible  to  a 
general  procedural  skill  alone.  Rather,  it  is  in  part  determined  by  a  large  body  of 
organized  knowledge  structured  for  the  purposes  of  solving  problems  in  a 


particular  domain  and  in  part  determined  by  a  set  of  quite  specific  rules, 
procedures,  or  heuristics  which  act  upon  that  knowledge. 

AI  research  also  has  much  to  contribute  to  understanding  genuine 
problem  solving  tasks,  in  particular,  expert  systems  technology.  Consider  those 
expert  systems  in  use  today  based  on  a  technique  known  as  backward  chaining. 
Basically,  expertise  is  captured  by  a  set  of  rules  of  the  form:  if  some  conditions), 
then  take  this  action.  In  backward  chaining,  the  initial  conditions  of  a  goal 
statement  are  examined,  and  rules  whose  actions  will  satisfy  those  conditions  are 
sought.  This  process  is  repeated  recursively  until  all  of  the  goal  condition  is 
satisfied.  An  explanation  of  this  process  is  provided  by  Waltz  (1982).  A  design 
goal  for  such  expert  systems  is  to  have  the  set  of  rules  which  are  comprehensible 
to  human  experts.  In  this  way,  the  human  expert  can  "understand"  how  the  expert 
system  arrived  at  a  particular  conclusion  or  answer  by  examining  the  chain  of 
rules  which  led  to  that  answer. 

MYCIN  (Shortliffe,  1976)  is  an  example  of  a  backward  chaining  expert 
system.  The  backward  chaining  control  structure  is  implemented  through  the 
interaction  of  two  mechanisms,  MONITOR  and  FINDOUT.  MONITOR'S  function  is 
to  verify  whether  or  not  a  rule  is  true.  To  do  this,  it  must  establish  the  truth 
value  of  each  of  its  conditions.  If  one  of  the  conditions  is  riot  true,  the  rule  fails 
and  its  actions  may  not  be  taken.  Often  it  is  not  possible  to  determine  the  truth 
value  of  one  of  the  conditions.  In  such  cases  the  FINDOUT  mechanism  is  called. 
FINDOUT  retrieves  the  list  of  rules  which  have  an  action  which  establishes  the 
truth  value  of  the  condition  in  question.  At  this  point,  the  problem  is  to  see  if  any 
of  these  rules  are  true.  In  order  to  do  this,  FINDOUT  calls  MONITOR  with  the 
task  of  examining  each  of  these  rules.  If  a  condition  in  one  of  these  rules  is 
indeterminate,  the  process  is  continued  working  backward  (e.g.,  FINDOUT  is 
called  again).  Backward  chaining  is  recursive  because  of  the  way  a  mechanism 
calls  itself  in  the  course  of  processing  (e.g.,  MONITOR  calls  FINDOUT  which  calls 
MONITOR,  etc.). 

Backward  chaining  systems  typically  do  not  explicitly  contain  models  of 
causality,  functionality,  or  structure  for  the  domain  under  consideration.  This  is  a 
limitation  because,  for  the  toughest  of  problems,  it  is  often  necessary  to  reason 
from  a  causal,  functional,  or  structural  model  of  the  domain  in  order  to  arrive  at 
a  solution.  These  and  other  issues  regarding  expert  systems  are  cogently 
addressed  in  a  paper  by  Randall  Davis  (1982). 

In  seeking  help  for  the  challenges  facing  the  Air  Force  in  the 
development  of  valid  and  useful  analyses  of  tasks  that  require  genuine  problem 
solving  skill,  slight  differences  in  orientation  between  the  psychological  and  AI 
oriented  approaches  must  be  considered.  In  psychological  models  of  problem 
solving,  comprehensiveness,  rigor,  and  theoretical  value  are  of  more  importance 
than  is  performance  capability.  The  opposite  is  true  of  AI  expert  systems, 
because  performance  is  of  paramount  interest,  and  at  times  certain  aspects  of 
performance  are  forced  or  glossed  over  in  ways  that  do  not  parallel  or  model 
human  performance,  for  example,  in  exhaustive  search.  Both  of  the  approaches  to 
genuine  problem  solving  have  value  to  the  Air  Force  in  enhancing  the 
methodologies  of  task  analysis  in  complex  domains.  The  psychological  approach  is 


important  because  the  task  analyses  will  be  applied  to  instruction;  ultimately, 
human  performance  is  the  subject  of  interest.  On  the  other  hand,  in  the  area  of 
JPAs,  it  may  be  permissible  to  have  an  expert  system  arrive  at  some  conclusion  or 
take  some  action  (such  as  a  curriculum  sequencing  decision)  that  is  not  entirely 
the  way  a  human  would  do  it. 


Training  in  the  Air  Force  is  provided  both  as  formal  instruction  and  on- 
the-job.  By  far  the  greatest  formal  training  emphasis  is  on  in-residence  training. 
For  the  Military  Airlift  Command  and  Strategic  Air  Command  there  is  a  trend 
toward  less  formal  schooling,  and  more  03T  at  the  unit  level.  For  ATC  the  trend 
is  toward  more  centralized  training  and  more  self-paced/CAI  training  (Rosenblum, 
1979).  CXJT  relies  almost  exclusively  on  local  unit  management  of  training  and 
certification  programs,  largely  on  an  informal  and  subjective  basis.  Challenges  in 
training  include: 

1.  The  need  for  valid,  quality,  useful  task  analyses. 

2.  The  need  to  present  instruction  that  is  a  proper  blend  of 
rote  learning  of  procedures  and  robust  learning  of 
relevant  principles. 

3.  The  need  for  efficient  and  timely  methods  of  instructional 
materials  development  and  revision. 

4.  The  need  of  effective  instructional  materials. 

5.  The  need  for  less  reliance  on  actual  equipment  as  the  sole 
source  of  practical  hands-on,  job-relevant  experience. 

Shortage  of  trained  personnel  is  a  good  organizing  principle  for 
considering  training  issues.  The  ultimate  source  of  shortage  is  personnel  attrition. 
Training  programs  produce  individuals  with  expertise  that  is  required  in  the 
private  sector;  the  Air  Force  often  acts  as  a  training  ground  for  industry.  In 
addition,  a  shortage  of  expertise  results  from  the  continuing  evolution  in  the 
complexity  of  Air  Force  weapons  systems.  This  generates  a  demand  for  personnel 
who  have  the  skill  to  understand  and  work  with  systems  that  are  increasingly 
complex  and  technical. 

This  issue  of  expertise  shortage  is  at  the  heart  of  a  conflict  between  the 
training  and  field  missions  within  the  Air  Force.  The  field  feels  a  shortage  of 
trained  personnel  and  would  like  the  graduates  of  the  training  programs  to  be 
trained  to  a  higher  technical  caliber.  But  producing  this  caliber  robs  the  field  of 
expertise  (in  the  form  of  instructors)  and  of  useful  duty  time  (in  the  form  of  extra 
time  spent  in  training).  The  Air  Force  needs  to  maximize  in-residence  training 


graduates’  competence  while  minimizing  the  amount  of  resources  spent.  A 
solution  to  this  problem  is  to  increase  training  effectiveness  both  in-residence  and 
especially  03T.  Of  course,  this  has  long  been  a  goal  for  the  Air  Force,  and 
progress  toward  this  goal  is  continuously  being  made. 

The  allocation  of  training  objectives  differentially  to  in-residence  and 
03T  programs  is  currently  done  on  a  formal  basis.  An  AFHRL  planned  Training 
Decisions  System  is  designed  to  act  as  a  management  aid  in  making  these 
allocation  decisions,  decisions  which  have  a  large  potential  impact  on  overall 
training  effectiveness  and  efficiency  within  the  Air  Force. 

AI  techniques  can  provide  approaches  to  increasing  training 
effectiveness.  The  approaches,  suggested  below,  may  be  applied  to  either  the  in¬ 
residence  or  on-the-job  training  environments.  Because  in-residence  training 
emphasizes  classroom  sized  groups  of  students,  applications  of  instructional 
technology  may  be  most  appropriate  in  the  03T  environment  where  training  is 
more  structured  around  the  individual  than  in  in-residence  training.  The  needs 
identified  previously  are  found  in  both  in-residence  and  03T  environments,  and 
approaches  suggested  in  the  following  paragraphs,  in  appropriate  mixes,  will  be 
suitable  in  both  of  these  environments. 


Approaches 

Task  analysis.  AI  applications  in  support  of  this  foundational  component 
of  instructional  systems  design  have  already  been  treated  in  detail.  Successful 
progress  in  areas  of  identified  need  in  training  depend  on  front-end  work  in  task 
analysis.  The  task  analysis  does  not  satisfy,  in  and  of  itself,  any  pressing  training 
need.  But,  application  and  we  of  task  analyses  are  of  fundamental  importance  in 
instruction.  In  the  remaining  subsections,  particular  training  applications  of  task 
analyses,  the  cognitive  psychology  of  problem  solving,  and  work  in  knowledge 
engineering  and  expert  systems  are  presented. 

Selection  of  the  appropriate  mix  of  rote  learning  and  general  principles 
learning.  This  is  an  old  and  familiar  issue,  and  some  have  observed  that  the 
current  orientation  toward  this  issue  swings  from  pole  to  pole  cyclically.  As  old 
and  timeworn  as  this  issue  is,  recent  advances  in  cognitive  science  provide  fresh 
insights  into  the  problem  of  rote  versus  meaningful  instruction. 

The  Kieras  and  Poison  formalism  previously  discussed  provides  a 
straightforward  approach  to  this  problem.  They  have  developed  a  set  of  criteria 
for  deciding  when  "how-it-works"  knowledge  should  be  useful  to  novices.  Briefly, 
these  criteria  are  (a)  if  the  user's  task  representation  contains  a  goal  that  is 
unique  to  the  device  the  user  is  working  with  and  not  previously  encountered  in 
different  devices,  then  that  goal  should  be  supported  with  an  explanation  derived 
from  how-it-works  knowledge,  (b)  if  a  goal  in  the  task  representation  is  already  a 
part  of  the  user's  expected  standard  repertoire  of  performance  capability,  that 
goal  need  not  be  supported  with  explanation,  and  (c)  if  how-it-works  knowledge 
does  not  correspond  with  a  particular  goal  in  the  user's  task  representation,  then 
that  explanation  should  not  be  provided.  These  rules  are  familiar;  the  second  one 
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addresses  the  entering  behaviors  consideration  in  instructional  design,  and  the 
third  is  a  tenet  of  functional  context  training  (Shoemaker,  1960).  What  the  Kieras 
and  Poison  formalism  contributes  is  a  formal  descriptive  system  on  which  to  base 
these  rules. 

Much  work  has  been  done  on  meaningful  versus  rote  learning,  beginning 
with  classic  studies  by  Brownell  (1935).  More  recently,  the  work  of  Mayer  (Mayer, 
1974;  Mayer  &  Greeno,  1972;  Mayer,  Greeno  &  Stiehl,  1975)  sheds  light  on  the 
importance  of  this  problem.  The  findings  from  these  experiments  on  rote  versus 
meaningful  instruction  for  a  variety  of  tasks  indicate  that  qualitative  differences 
in  learning  outcome  are  found  as  a  function  of  instructional  treatment.  During  an 
instructional  treatment,  students  will  selectively  attend  to  particular  aspects  of 
the  presentation  in  accordance  with  the  learning  "set"  activated  by  the  treatment. 
If  students  expect  instruction  to  be  rote,  then  they  learn  by  rote.  If  they  expect 
meaningful  instruction,  then  they  attend  to  meanings.  The  outcome  of  a 
meaningful  approach  is  that  students  are  more  capable  of  solving  novel  problems, 
but  solve  familiar  problems  less  quickly  than  do  students  trained  with  more 
emphasis  on  rote  learning  of  fixed  procedures. 

Overall,  work  in  the  psychology  of  expertise  yields  the  result  that,  once 
again,  performance  is  largely  a  function  of  command  of  and  access  to  large 
amounts  of  knowledge.  It  has  been  estimated  that  expertise  in  chess  (and  perhaps 
expertise  in  numerous  fields)  may  be  accounted  for  by  some  50,000  "chunks"  of 
knowledge  which  require  upwards  of  10  years  to  acquire.  If  this  is  the  case,  then 
even  though  meaningful,  principles-oriented  instruction  promotes  a  certain 
robustness  in  learning  and  memory,  there  is  no  substitute  for  experience.  The 
training  implications  of  this  finding  are  that  in  performance  areas  where  expertise 
is  needed  (as  in  genuine  problem  solving  tasks),  one  should  not  expect  too  much 
from  training.  If  expertise  is  in  critical  shortage,  JPAs  may  be  a  more  realistic 
solution  than  beefed-up  training  programs.  This  is  further  treated  in  the  section 
below  on  JPAs. 

One  of  the  most  interesting  findings  of  the  problem  solving  literature 
(cited  above)  is  that  novice  and  expert  performance  can  differ  qualitatively  in 
reaching  the  correct  solution  to  a  given  problem.  The  explanation  for  this  lies  in 
the  different  representations  and  search  methods  employed  by  novices  versus 
experts.  Misconceptions  in  student  understanding  (McCloskey,  1982;  Stevens, 
Collins  &  Goldin,  1978)  further  contribute  to  the  distinction  between  novice  and 
expert.  It  has  become  clear  that  protocol  analyses  of  novices  (e.g.,  task  analyses 
from  the  novice's  point  of  view)  are  critically  important  in  understanding  how  to 
approach  instruction.  It  is  not  sufficient  to  expect  novices  to  learn  the  expert 
approach.  First,  the  misconceptions  must  be  dismantled,  and  then  a  transitional, 
developmental  progression  from  novice  to  expert  must  be  supported  with 
instruction.  This  realization  is  at  the  root  of  an  approach  to  the  use  of  an  expert 
system  as  the  basis  of  a  tutorial  system  (Clancey,  1979;  Clancey  &  Letsinger, 
1981).  The  rules  in  an  expert  system  are  tightly  "compiled"  pieces  of  knowledge. 
They  contain,  implicitly,  structural  and  procedural  knowledge  that  must  be  made 
available,  explicitly,  to  the  student.  In  addition,  justifications  for  the  rules  must 
be  available  to  the  student.  Justifications  serve  as  memory  aids  for  remembering 
the  rules  and  as  a  basis  for  deep  understanding  that  allows  students  to  learn 
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beyond  what  is  captured  in  the  rules.  Expertise  cannot  be  transmitted  intact;  a 
lot  oi  recompilation  of  basic  principles,  under  the  guidance  of  experience,  is 
needed  before  fresh  expertise  can  be  developed. 

Buildin£  blocks  and  tools  for  conventional  instruction  and  CAL  NPRDC 
has  launched  the  Authoring  Instructional  Materials  (AIM)  project  focused  on  using 
computers  to  automate  and  assist  curriculum  development  for  both  computer- 
delivered  and  conventional  instruction.  TAEG  has  underway  the  Computer  Aided 
Authoring  System  project  with  the  objective  of  developing  an  authoring  system 
capable  of  generating  job  aids,  learning  aids  (procedural  and  nomenclature), 
curriculum,  instructor  guides,  and  student  guides. 

There  currently  exists  a  number  of  authoring  and/or  curriculum 
development  systems,  building  blocks,  tools,  or  aids,  for  example,  the  Writer's 
Workbench  at  Bell  Labs,  the  Computer  Readability  Editing  System  (CRES)  at 
TAEG,  the  Computer-Aided  System  for  Developmental  Aircrew  Training 
(CASDAT)  at  NTEC,  Computerized  Job  Aids  for  ISD  at  ARI  and  the  Human 
Resources  Research  Organization  (HumRRO),  student  flow  simulation  for  course 
design  at  the  Air  Force  and  Rand,  and  commercially  available  authoring  systems 
for  computer-based  instruction  available  from  such  companies  as  WICAT,  Bell  & 
Howell,  Control  Data  Corporation,  Hazel  tine,  etc. 

An  emphasis  or  reliance  on  isolated  building  blocks  and  tools  for 
designing  and  developing  instruction  may  not  be  the  best  approach  to  the  problem 
of  developing  conventional  CAI.  A  standard  way  of  conceiving  of  computer-based 
instruction  software  systems  is  to  see  text  and  graphics  (e.g.,  display-oriented 
editors)  as  a  basic  building  block,  which  is  encapsulated  within  an  instructional 
interaction  editor  used  to  build  lessons  (e.g.,  a  CAI  editor),  which  is  in  turn 
encapsulated  within  a  curriculum  management  editor  controlling  the  progress  of 
students  through  sets  of  lessons  constituting  courses  (e.g.,  a  CMI  system).  Many 
systems  exist  that  are  structured  in  this  way,  c.f.s  WICAT,  AIS,  PLATO,  etc. 
While  these  systems  do  provide  structures  to  be  filled  in  at  various  levels  (course 
hierarchies,  objectives,  and  instructional  interactions),  there  is  little  or  no  help 
and  assistance  for  the  instructional  designer  in  making  design  decisions,  such  as 
which  objectives  to  select,  how  to  structure  and  organize  them,  how  to  develop 
effective  feedback,  or  how  to  build  adaptive  branching  schemes.  It  is  suggested 
that  this  is  an  area  where  AI  might  be  applied. 

The  effectiveness,  efficiency,  and  timeliness  of  courseware  design, 
development,  and  delivery  can  be  increased  by  integrating  these  three  "Ds"  into 
one  coordinated  system.  At  AFHRL,  the  Interactive  Graphics  Simulation  (1GS) 
project  has  in  place  an  integrated  design,  development,  and  delivery  system  for 
task-oriented  avionics  maintenance  simulations.  Control  Data  Corporation  is 
currently  placing  an  emphasis  on  this  approach. 

The  key  to  the  IGS  approach  lies  in  the  interactive  development, 
refinement,  and  modification  of  a  hierarchical  task  analysis  of  the  job.  This  is 
currently  accomplished  on-line  on  the  AIS  through  use  of  specially  designed 
editors.  The  resultant  on-line  task  hierarchy  is  then  used  to  develop  simulation 
scenarios,  complete  with  scene-setting  text  and  graphics,  video  disc  control 


commands,  and  feedback  messages.  During  delivery,  the  task  hierarchy  is  used  to 
represent  the  alternatives  available  to  the  problem  solver  at  each  step  of  the  way 
toward  a  solution.  Since  correct  and  incorrect  choices  are  nodes  in  the  task 
hierarchy,  a  record  of  a  given  student's  responses  across  a  number  of  simulations 
provides  a  model  of  the  student's  competence  in  terms  of  the  task  hierarchy.  This 
model  is  then  used  to  control  instructional  interaction  toward  the  goal  of  focusing 
student  attention  on  unmastered  portions  of  the  task.  For  example,  when  a 
student's  response  history  indicates  that  the  student  has  mastered  a  commonly 
reoccurring  subtask,  that  subtask  is  summarized,  thus  freeing  up  time  and 
attentional  resources  for  subtasks  that  do  need  further  practice.  Explanations  and 
illustrations  of  this  integrated  design,  development,  and  delivery  system  are 
presented  in  Dallman  (1982),  Richardson  (1982)  and  Pieper  (1982). 

The  key  to  the  success  in  developing  such  an  integrated  system  was  the 
pivotal  use  of  an  on-line  task  analysis  representation.  As  emphasized  in  earlier 
sections  on  task  analysis,  a  task  analysis  approach  to  integrating  systems  can  be 
extended  beyond  adaptive  drill  and  practice  (which  is  what  the  interactive 
graphics  simulation  system  is)  to  include  explanatory  instruction,  performance 
measurement,  and  job  performance  aiding  as  well.  When  systems  are  closely 
integrated  through  the  use  of  task  analyses,  this  facilitates  courseware 
maintenance  and  helps  establish  courseware  validity. 

Intelligent  CAL  Intelligent  CAI  (ICA I)  offers  a  way  of  enhancing  the 
effectiveness  of  interactive  computer-based  instruction  beyond  what  is  achievable 
with  traditional  approaches  to  tutorial,  drill  and  practice,  and  simulation 
courseware.  I CAI  seeks  to  emulate  the  professional  competence  of  good  teachers 
when  working  one-on-one  with  a  student.  Good  teachers  (a)  know  what  they  are 
doing,  i.e.,  have  subject  matter  competence;  (b)  are  vitally  concerned  with 
understanding  the  state  of  their  students'  knowledge,  including  misconceptions; 
and  (c)  have  a  good  "bag  of  tricks"  concerning  pedagogical  approaches,  for 
example,  the  good  teacher  knows  that  the  student  needs  a  proper  mix  of 
instruction  and  practice.  AI  in  general  tries  to  emulate  the  actions  of  intelligent 
agents,  and  ICAI  attempts  to  emulate  intelligent  teachers. 

The  main  components  of  an  ICAI  system  are  problem  solving  expertise, 
the  student  model,  and  tutorial  strategies.  An  exhaustive  review  of  the  ICAI 
literature  is  beyond  the  scope  of  this  paper.  Integrative  reviews  of  this  field  have 
appeared  in  the  literature,  and  include  those  by  Gable  and  Page  (1980),  Barr  and 
Feigenbaum  (1982),  and  Sleeman  and  Brown  (1982). 

Table  2  briefly  summarizes  a  number  of  ICAI  systems,  indicating 
literature  references  and  subject  matter  nature.  Not  all  of  the  three  components 
of  an  ICAI  system  are  fully  implemented  in  every  system.  Each  component  by 
itself  is  a  difficult  problem,  and  the  successful  implementation  of  a  demonstration 
program  often  depends  on  strategic  decisions  regarding  where  to  try  to  work  at 
the  state  of  the  art,  and  where  to  settle  for  well-known  approaches.  For  example, 
the  more  effort  that  has  to  be  put  into  explicating  domain  expertise,  the  less 
resources  are  available  for  developing  student  modeling  and  pedagogical  modules. 
ICAI  systems  consist  of  three  main  components,  subprograms,  or  modules.  That 
is,  at  the  highest  level  of  program  organization,  an  ICAI  system 
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Table  2.  Intelligent  CAI  Systems 


SYSTEM 

REFERENCES 

SUB3ECT  MATTER  CONTENT 

SCHOLAR 

Carbonell  (1970) 

Collins  (1976) 

South  American  geography,  text 
editing  systems 

WHY 

Stevens,  Collins  6c  Goldin  (1978) 
Stevens  6c  Collins  (1978) 

extension  of  SCHOLAR 

SOPHIE 

Brown,  Burton  6c  de  Kleer  (1982) 

electronics  troubleshooting 

WEST 

Burton  Sc  Brown  (1979) 

elementary  arithmetic  game 

WUMPUS 

Carr  Sc  Goldstein  (1977) 

Goldstein  (1977,  1979) 

Stansfield,  Carr  <5c  Goldstein 
(1976) 

fantasy  dungeons  and  dragons  game 

GUIDON 

Clancey  (1979) 

Clancey  6c  Letsinger  (1981) 

infectious  blood  disease 

BUGGY 

Brown  Sc  Burton  (1978) 

Brown  6c  VanLehn  (1980) 

elementary  subtraction 

EXCHECK 

Blaine  6c  Smith  (1977) 

Smith,  Graves,  Blaine,  6c 

Marinov  (1975) 

Smith  6c  Blain  (1976) 

Suppes (1981) 

logic,  set  theory,  proof  theory 

BIP 

Barr,  Beard  6c  Atkinson  (1976) 

BASIC  programming 

ACTS 

Hopf-Weichel  et  al.  (1980) 

electronics  troubleshooting 

STEAMER 

Stevens  et  al. 

(1982) 

propulsion  engineering 

ADA  TUTOR 

Computer* Thought  Corp.  (1982) 

Ada  programming 

consists  of  an  expert  module,  a  student  model  module,  and  a  tutor  module.  A  very 
brief  summary  of  the  three  ICAI  components  follows. 


The  expert  module.  The  expert  module  serves  to  generate  and  solve 
problems.  The  first  role  for  this  module  is  the  generation  of  problems.  This 
function  is  closely  linked  with  the  student  model  and  pedagogical  modules  because 
curricular  decisions  involve  what  the  student  knows  or  does  not  know  and 
strategic  decisions  about  what  to  present  next  and  how  to  present  it.  The  degree 
to  which  ICAI  systems  generate  curricula  of  problems  in  concert  with  student 
modeling  and  pedagogical  considerations  is  very  limited.  All  systems  do  have  a 
scheme  for  sequencing  problems,  and  approaches  vary.  The  second  role  for  the 
expertise  module  is  in  solving  problems.  This  is  done  in  order  to  evaluate  and 
critique  student  solutions.  ICAI  systems  with  "articulate"  problem  solvers  can 
explain  how  they  reached  a  solution  or  how  they  would  like  the  student  to  try  and 
reach  a  solution.  If  the  problem  solver  is  not  articulate,  what  is  termed  "opaque," 
then  the  system  can  offer  no  detailed  critique  of  a  student's  solution  path.  The 
extent  to  which  ICAI  systems'  problem  solvers  are  articulate  or  opaque  also 
varies. 


The  student  module.  The  objective  of  the  student  modeling  module  is  to 
understand  what  curricular  objectives  the  student  has  mastered,  and  to  understand 
or  have  representations  for  predictable  misconceptions  and  suboptimal 
approaches.  Input  to  the  student  model  may  be  derived  from  numerous  sources 
including:  (a)  a  differential  comparison  of  the  student's  behavior  and  the  behavior 
output  of  the  expert  module  on  a  given  problem  or  question,  (b)  explicit 
information  derived  from  direct  questions  asked  of  the  student,  and  (c)  historical 
assumptions  based  on  the  student's  experience.  The  student's  knowledge  can  be 
represented  as  an  overlay  or  subset  of  the  expert's  knowledge,  or  it  can  be 
represented  as  deviations  from  the  expert's  knowledge  (e.g.,  misconceptions, 
suboptimal  approaches,  and  "buggy"  procedures).  This  is  a  very  important 
distinction.  As  was  explained  in  a  previous  section  on  the  difference  between  rote 
and  meaningful  instruction,  expertise  is  not  something  that  can  be  transferred 
directly  to  a  novice.  There  is  a  developmental  process  that  must  be  gone  through 
as  the  backward  chaining  approach  typical  of  "uncompiled"  novice  competence  is 
refined  (through  practice,  experience,  and  more  knowledge),  into  the  pattern 
matching,  methods  application,  forward  reasoning  character  of  expert 
performance.  To  represent  and  use  a  genuine  expert  representation  for  the  expert 
in  an  ICAI  system  may  be  a  mistake,  as  was  discovered  by  Clancey  in  his 
experiments  with  MYCIN  (Clancey  &  Letsinger,  1981).  A  deeper  representation  is 
needed  in  order  that  differences  detected  between  the  student's  performance  and 
that  of  the  experts  are  to  be  minimized.  The  purposes  or  plans  a  student  is  using 
must  be  inferred  from  the  student's  actions.  Therefore,  "plan  recognition"  and 
plan  representation  are  of  great  importance  in  ICAI  systems  R&D. 

The  tutor  module.  The  third  module  is  the  tutor  module.  It  is  a  well- 
known  fact  that  simply  because  someone  has  complete  subject  matter  mastery, 
his  or  her  ability  to  transmit  that  mastery  does  not  necessarily  follow.  There  is  a 
great  deal  of  procedural  knowledge  involved  in  effective  instruction.  This 
knowledge  comprises  the  basis  for  the  tutoring  strategies  module  of  the  ICAI 
system.  Pedagogical  knowledge  of  curriculum  sequencing  is  important,  for 
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example  in  selecting  an  instructive  problem,  tailor-made  to  help  students  see  a 
point  for  himself.  Knowledge  concerning  how  to  conduct  a  Socratic  dialog  may 
also  form  part  of  the  knowledge  data  base  of  the  tutoring  module.  The  Socratic 
dialog  is  a  very  refined  version  of  discourse.  Hence,  modern  cognitive  theories  of 
discourse  comprehension  and  production  have  much  to  contribute  to  the 
development  of  tutoring  systems.  For  example: 

Question  a  prediction  made  without  enough  information. 

If:  a  student  makes  a  prediction  as  to  the  value  of  the 
dependent  variable  on  the  basis  of  some  set  of  factors, 
and  there  is  another  value  consistent  with  that  set  of 
factors, 

Then:  ask  the  student  why  not  the  other  value.  (Collins, 

1978,  p.3). 

Other  ICAI  issues.  Other  issues  of  concern  to  intelligent  CAI  besides  the 
three  basic  ICAI  modules  include  (a)  student-system  interface  issues,  e.g.,  the  use 
of  natural  language  and  all  of  the  problems  associated  with  generating  and  parsing 
prose  text,  (b)  self-improving  tutors,  e.g.,  ICAI  systems  that  learn  how  to  improve 
their  instructional  effectiveness,  based  on  experience,  and  (c)  the  use  of  graphics 
as  a  medium  of  communication.  The  natural  language  problem  is  a  major  area  of 
A1  research  in  its  own  right,  and  it  is  certainly  not  solved.  Means  exist  to  provide 
for  natural  language  interaction  between  student  and  system,  but  they  are  limited 
and  must  be  treated  carefully.  Again,  there  is  a  resource  tradeoff  between 
emphasis  on  this  and  other  areas  of  importance  in  ICAI  systems.  Very  little  R&D 
on  self-improving  systems  has  been  conducted.  The  achievement  of  adequate 
baseline  performance  is  still  a  demanding  challenge.  (See  Sleeman  &  Brown, 
1982).  The  presentation  to  the  student  of  imagery  in  instructional  systems  seems 
to  be  simply  a  matter  of  broadening  the  communications  channel  between  system 
and  user  and  offers  no  fundamental  technical  challenges.  Having  the  computer 
"see"  images  drawn  by  the  student  is,  however,  not  yet  possible. 

ICAI  systems  to  date  have  been  "hand  crafted"  and  have  usually  focused 
on  less  than  the  totality  of  issues,  concerns,  and  components  of  an  ICAI  system. 
This  suggests  that  practical  AI  applications  must  not  try  to  press  the  state  of  the 
art  on  all  fronts.  Practical  efforts  should  focus  on  discrete,  well  defined,  and  well 
understood  content  areas  that  are  priority  training  areas.  Computer  programming 
is  a  candidate,  but  its  task  domain  is  not  yet  fully  understood.  Troubleshooting  is 
a  generic  area  that  seems  to  hold  great  promise  as  a  candidate  applications  area, 
especially  because  it  is  a  priority  training  need.  Much  general  work  on  fault 
diagnosis  and  isolation  has  been  done;  the  area  telescopes  nicely  from  general 
troubleshooting  procedures,  to  electronic  troubleshooting,  to  troubleshooting 
specific  systems.  The  ACTS  system  (Hopf-Weichel,  Purcell,  Freedy,  ic  Lucaccini, 
1980)  and  the  SOPHIE  system  (Brown,  Burton,  ic  de  Kleer,  1982)  are  tw'* 
completed  ICAI  projects  in  electronics  troubleshooting.  Several  current  DoL 
research  projects  are  continuing  research  in  troubleshooting.  See  Chapter  II. 

For  ICAI  to  be  of  practical  use  in  the  training  mission  of  the  Air  Force,  a 
fully  developed,  robust  system  is  needed.  Demonstration  projects  will  have  little 


utility  in  the  production  and  delivery  environment.  Systems  must  be  designed  to 
be  used,  not  to  serve  as  research  vehicles  or  demonstrate  state-of-the-art  ICAI 
techniques.  This  means  that,  as  a  part  of  the  design  and  development  of  a 
practical  ICAI  system,  the  field  and  school  must  have  considerable  input  from  the 
outset.  This  approach  is  being  used  by  the  STEAMER  project  at  NPRDC,  where 
interim  field  tryouts  are  now  underway. 

The  fact  that  ICAI  systems  are  currently  hand  crafted,  and  that  the  first 
few  applications  environment-oriented  systems  will  also  be  hand  crafted,  should 
not  deter  a  program  focused  on  the  development  of  tools  that  will  be  useful  in 
moving  toward  a  mass  production  capability  for  ICAI  systems.  Recall  that  the 
design  and  development  process  for  traditional  hard  copy  and  CAI  materials  is 
still  largely  an  art.  The  techniques  are  well  specified,  as  in  the  ISD  process,  but 
little  or  no  automated  design  help  is  yet  available. 

Task  analysis.  The  techniques  reviewed  above  for  automated  assistance 
to  the  design  and  development  of  traditional  materials,  especially  the  central  role 
of  the  task  analysis,  will  probably  have  a  payoff  for  the  design  and  development  of 
ICAI  systems  as  well.  In  particular,  a  large  effort  focused  on  developing  a 
"telescoping  task  hierarchy"  for  troubleshooting  and  fault  diagnosis  tasks  might 
serve  as  the  first  step  toward  design  assistance  in  CAI  and  ICAI  as  well.  By 
"telescoping"  is  meant  a  representation  that  is  capable  of  representing:  the 
troubleshooting  processes  involved  in  a  particular  system  (e.g.,  Fill  6883 
Avionics  Test  Station);  which  also  serves  to  outline  the  troubleshooting  process  at 
the  next  level  of  generality  (e.g.,  avionics  test  stations);  and  so  on,  to  the  most 
general  level  of  detail  (e.g.,  general  troubleshooting  strategies,  approaches,  and 
procedures). 

Building  blocks  and  tools.  Specialized  building  blocks  and  tools  for  a 
particular  applications  area  (e.g.,  troubleshooting  or  computer  programming)  may 
be  necessary.  Natural  language  interfaces,  pattern  matchers,  plan  recognizers, 
production  system  interpreters,  augmented  transition  network  compilers,  and 
graphics  editors  and  graphics  generators  are  examples  of  more  general  purpose 
building  blocks  and  tools.  A  carefully  thought-through  building  block  development 
project  should  parallel  the  first  few  ICAI  systems  development  projects.  Given 
the  centrality  of  task  analysis,  automatic  or  semi-automatic  means  of  "knowledge 
extraction"  become  important,  for  a  task  analysis  is  an  explication  of  a  subject 
matter  expert's  expertise.  The  AFHRL  Interactive  Graphics  Simulator  project  has 
demonstrated  a  technology  for  semi-automatic  knowledge  extraction,  which  is  an 
integrated  part  of  an  instructional  materials  design,  development,  and  delivery 
system.  Finally,  instructional  design  itself  is  a  target  area  for  knowledge 
extraction,  for  the  procedures  inherent  in  the  design  of  good  instruction  play  an 
important  role  in  the  tutor  module  of  ICAI  systems.  Certain  canonical  approaches 
to  instruction  may  be  identified,  and  their  procedural  aspects  codified  in  on-line 
representations. 

Use  of  graphics.  Graphical  or  pictorial  imagery  (objects,  structure,  or 
process)  are  a  major  means  of  communication.  Imagery  is  so  important  in  mental 
processing  that  there  is  great  debate  in  cognitive  psychology  as  to  whether 
encoding  in  long-term  memory  is  in  terms  of  propositions,  images,  or  perhaps  both 
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(Pylyshyn,  1973).  However,  when  a  technician's  job  involves  working  with  actual 
objects  and  pictorial  representations  of  such  objects,  graphical  or  pictorial 
capabilities  in  instructional  systems  are  necessary. 

The  recent  advent  of  the  video  disc  and  recent  improvements  and 
accessibility  of  computer  generated  graphics  systems  make  the  incorporation  of 
imagery  into  instructional  systems  practical.  Indeed,  it  may  even  be  possible  to 
replace  large  assemblages  of  equipment,  in  particular  electronics  equipment,  with 
graphics  simulations.  This  has  important  implications  for  training.  More  hands-on 
practice  can  be  provided  (albeit  simulated)  than  might  otherwise  be  possible  with 
expensive  physical  hardware.  Also,  since  these  graphics  systems  are  designed 
from  scratch,  it  is  possible  to  design  them  for  the  purposes  of  training,  such  that 
hour  for  hour,  more  learning  may  go  on  through  interaction  with  a  graphics 
training  device  than  with  the  actual  equipment.  These  issues  are  now  being 
studied  in  numerous  DoD  research  projects. 

Graphics  or  imagery  CBE  systems  are  functionally  identical  to  CBE 
systems  in  general.  The  same  distinctions  between  frame-oriented  CAI,  drill  and 
practice,  and  simulation  apply.  The  same  opportunities  for  ICAI  exist.  Graphics 
and  imagery  simply  provide  an  additional  modality  of  communication.  This  is  an 
incredibly  powerful  and  important  modality;  therefore,  it  should  be  exploited  to 
the  maximum.  Approaches  to  the  development  of  quality  interactive  instruction 
in  general  apply  to  the  development  of  quality  video  disc  or  graphics-based 
instruction  in  particular.  Design  and  development  problems  should  be  attacked  at 
the  most  general  level  possible,  e.g.,  at  the  level  of  "interactive  instruction"; 
effort  need  not  be  duplicated  by  addressing  fundamentally  the  same  issues  at  the 
more  specific  level  of  detail  of  "graphics  simulations."  Therefore,  the  approaches 
to  design,  development,  and  delivery  suggested  in  preceding  sections  apply  to 
highly  graphics-oriented  systems  as  well. 


Performance  Measurement 


Challenges 

Roles  of  performance  measurement  in  the  military  include  job 
proficiency  assessment,  training  effectiveness  assessment,  personnel  selection  and 
classification,  certification,  promotion,  and  research  and  development.  The  focus 
of  job  proficiency  assessment  may  be  on  the  individual,  team/crew,  unit,  service, 
or  military-wide  readiness.  The  roles  or  purposes  of  performance  measurement 
define  different  mixes  of  performance  tests  and  administrators.  For  example, 
OJT  performance  measurement  may  be  based  on  very  informal  and  subjective 
judgments  on  the  part  of  a  shop  supervisor,  who  needs  to  know  whether  an  airman 
can  perform  a  certain  task  indicated  in  an  Air  Force  Job  Proficiency  Guide.  This 
approach  differs  greatly,  for  example,  from  the  performance  measurement 
techniques  and  data  employed  by  the  R&D  community,  where  data  requirements 
are  more  rigorous,  or  needed  in  norm  rather  than  criterion  referenced  formats. 
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Career  specialty  characteristics  also  influence  applicable  performance 
measurement  techniques  and  procedures.  For  example,  career  specialties  with 
critical  tasks,  or  tasks  associated  with  high  safety  risks  or  high  costs,  require 
close  specialties  monitoring  through  performance  measurement  procedures.  Each 
Air  Force  base  now  has  a  quality  control  group  whose  duty  it  is  to  monitor  certain 
critical  career  specialties. 

Technical  career  specialties,  especially  maintenance,  require 
performance  monitoring  because  modern  weapons  systems  are  technologically 
advanced.  These  systems  are  complex  and  maintenance  is  often  far  from 
straightforward,  requiring  considerable  troubleshooting  skill.  Ineffective 
maintenance  practices  cam  cost  the  Air  Force  exhorbitant  funds  in  erroneously 
replaced  units.  Moreover,  mission  preparedness  is  vitally  linked  to  adequate 
maintenance  capability.  The  Air  Force  uses  about  28  percent  of  its  workforce  and 
between  $5  and  $7  billion  amnually  on  maintenance  (Townsend,  1980). 

Excessive  maintenance  costs  can  be  reduced  if  job  task  performance 
tests  or  empirically  valid  symbolic  substitutes  to  them  are  available  to  ascertain 
how  efficiently  and  effectively  maintenance  personnel  perform  their  tasks  (Foley, 
1977).  Performance  measurement  is  especially  feasible  in  maintenance  areas 
because  of  the  technical  nature  of  these  tasks.  Performance  is  the  result  of  a 
structured  application  of  knowledge,  not  nuance  or  intuition.  In  other  words,  it  is 
possible  to  obtain  good  definitions  of  the  goals  and  methods  of  performance  in 
these  areas.  The  task  analytic  approaches  discussed  previously  can  be  used  to 
develop,  specif iy,  and  sample  specific,  concrete,  performance  measures.  The 
hierarchical  nature  of  task  analyses  is  useful  in  developing  sampling  strategies. 

The  remainder  of  this  section  is  focused  on  individual  job  proficiency 
assessment  in  technical  career  specialties,  especially  maintenance.  Measures  of 
individual  job  proficiency  may  be  used  in  turn  as  inputs  to  training  program 
assessment. 

During  and  after  World  War  II,  the  Air  Force  made  extensive  use  of 
formal  job  task  performance  tests.  For  reasons  of  economy,  these  tests  were 
abandoned  in  favor  of  paper-and-pencil  theory  and  job  knowledge  tests. 
According  to  Foley  (1977),  "none  of  these  substitute  measures  are  sufficiently 
valid  for  use  as  substitutes  for  job  task  proficiency  tests."  This  is  based  on  low 
correlations  obtained  between  job  task  performance  tests,  theory  tests,  job 
knowledge  tests,  and  school  marks. 

There  are  a  variety  of  job  proficiency  test  formats.  Some  numeric 
scores  for  performance  (for  example,  number  of  problems  correct)  have  no 
interpretive  value.  The  only  useful  presentation  of  job  performance  data  must  be 
some  meaningful  profile  of  proficiency.  Foley  has  developed  an  exemplar  of  such 
a  profile.  Performance  is  broken  down  into  various  functional  areas:  use  of  test 
equipment,  repair,  remove/install,  adjust/align/calibrate,  troubleshoot,  and 
checkout.  Hierarchical  dependencies  between  these  various  maintenance 
functions  are  identified.  Special  test  equipments  use  is  seen  as  a  component 
function  of  adjusting,  aligning,  and  calibrating  equipment.  The  adjust/align/ 
calibrate  functions  are,  in  turn,  seen  as  component  functions  of 
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troubleshooting.  In  this  formalism,  troubleshooting  is  explicitly  identified  as  the 
highest  order  function. 


Job  performance  tests  require  training  and  on-the-job  time.  Valid 
symbolic  substitutes  to  formal  hands-on  job  task  performance  measurement  may 
reduce  assessment  costs.  Paper-and-pencil  symbolic  representations  of  a  task 
have  potential  for  reducing  the  need  for  job  task  performance  testing  on  actual 
equipment.  Much  development  has  occurred  in  the  past  several  years  on 
simulation  systems  for  maintenance  training  (Orlansky  &  String,  1981).  Current 
developments  in  this  field  may  permit  the  use  of  maintenance  simulators  for 
performance  evaluation  and  certification  of  maintenance  personnel  on  an 
objective  basis  in  operational  environments. 

In  sum,  the  challenges  are  to  move  performance  measurement  into  the 
operational  environments  and  to  integrate  it  with  personnel  management  and 
training  organizations'  programs.  The  development  of  symbolic  substitutes,  based 
on  modern  maintenance  simulation  work  (such  as  interactive  graphics  simulations 
utilizing  intelligent  video  discs),  also  provides  a  challenge. 


Approaches 

The  development  of  valid  task  analyses  which  form  the  basis  for 
performance  measures  can  be  approached  utilizing  the  methods  and  techniques 
outlined  in  the  previous  section  on  task  analysis.  Since  troubleshooting 
proficiency  is  a  job  capability  of  interest,  task  representations  appropriate  to  the 
genuine  problem  solving  type  of  cognitive  skill  are  appropriate.  AI  techniques 
could  be  applied  to  assist  in  the  abstraction  from  a  number  of  sample  task 
protocols  of  a  generic  representation  for  the  task.  This  is  similar  to  the  "plan 
recognition"  capability  that  would  be  of  use  in  ICAI  systems. 

From  a  completed  task  analysis,  performance  test  problems  can  be 
constructed,  utilizing  the  hierarchical  nature  of  the  task  analysis  as  a  guide  in  a 
sampling  strategy  that  may  focus  on  criticality,  difficulty,  or  breadth  of 
coverage.  These  problems  could  be  administered  on  actual  equipment,  or  even 
developed  as  paper-and-pencil  tests.  But  by  far  the  best  medium  would  be 
interactive  graphics  simulation  so  that  the  data  collection  will  be  automatic, 
unbiased,  consistent,  and  immediately  incorporatable  into  an  integrated 
performance  assessment/training  system.  By  replacing  actual  equipment  with 
imagery  under  computer  control,  an  individual's  performance  protocol  can  be 
automatically  integrated  into  a  task  representation  used  to  characterize  that 
individual's  expertise.  A  problem  sequencing  component  may  be  used  to  prescribe 
a  sequence  of  OJT  practice  problems,  delivered  as  interactive,  instructional 
simulations,  to  remediate  specific  detected  inadequacies  in  the  subject's 
performance.  In  this  way,  the  next  tests  that  are  selected  for  administration  to 
the  subject  will  be  derived  from  a  representation  of  the  career  specialist's 
technical  competencies. 

As  mentioned,  traditional  job  proficiency  task  performance  tests  do 
require  on-the-job  time  and  resources.  The  approach  suggested  mitigates  against 
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this  problem  in  two  ways.  First,  special  attention  for  methods  of  task  sampling 
across  the  job  tasks  can  be  developed— automatic  methods  that  act  on  the  online 
job  representation.  In  this  way,  the  amount  of  performance  testing  may  be 
reduced.  Second,  since  performance  measurement  and  OJT  are  administered  on  a 
special  purpose  device,  current  demands  generated  by  OJT  and  assessment 
programs  on  supervisory  time  and  actual  equipment  are  alleviated. 


Job  Performance  Aiding 


Challenges 

Material  in  this  section  was  abstacted  from  Booher  (1978).  The  Booher 
report  is  a  survey  and  conceptual  analysis  of  existing  job  performance  aid 
techniques,  and  identifies  and  categorizes  factors  important  to  the  selection, 
design,  cost-performance  tradeoff,  conduct  of  future  RAD,  and  implementation  of 
performance  aiding  technology. 

Due  to  increasing  personnel  costs  and  declining  entry-level  skills,  there 
is  a  great  need  to  enhance  the  performance  of  lesser-trained  or  lesser-skilled 
personnel.  Other  RAD  projects  have  found  that  job  performance  aiding  has  major 
potential  for  increasing  job  performance  and  for  reducing  associated  costs. 
Analyses  reported  in  Serendipity  (1969)  and  Braid  (1975),  Rowan  (1973),  Shriver  and 
Hart  (1975),  and  Shriver  (1977)  indicate  that,  through  the  use  of  job  performance 
aiding  technology,  first-enlistment  training  costs  may  be  reduced  from  50  to  80 
percent,  a  significant  savings  in  spares  may  be  achieved  due  to  reductions  in 
erroneous  component  removal,  and  a  15  percent  decrease  in  the  maintenance  labor 
force  may  be  effected.  These  savings  total  $1,675  million  annually. 

Job  performance  aiding  liteature  has  a  long  history,  and  definitions  of 
JPA  go  back  as  far  as  1962.  Booher  (1978,  p.  3)  proposes  the  following  definition: 

A  job  performance  aid  is  any  device,  manual,  guide,  or  tool 

used  on  the  job  to  facilitate  performance  or  to  avoid  costs 

where  learning  is  incidental. 

Although  an  important  aspect  of  JPAs  is  the  fact  that  learning  is  incidental  and 
not  a  primary  product  of  their  use,  it  is  possible  to  design  aids  that  encourage 
greater  incidental  learning.  The  interrelationship  between  JPAs  and  training  is 
treated  in  the  final  section  of  this  chapter. 

The  types  of  generic  JPAs  include  troubleshooting,  nontroubleshooting, 
and  various  other  types:  standard  operating  procedures,  periodic  action  aids,  time 
critical  or  hazardous  conditions  aids,  one-trial  learning  aids,  and  aids  for  highly 
complex  integrated  systems.  In  addition  to  types  of  JPAs,  there  are  also  levels  of 
aiding.  Troubleshooting  aids  vary  from  fully  proceduralized  JPAs,  to  partially 
proceduralized  aids,  to  simple  logic  deductive  aids,  to  systems  descriptions. 
These  represent  a  continuum  from  prescriptive  aids  (which  tell  the  technician 
exactly  what  to  do  each  step  along  the  way  depending  on  the  exact  circumstances 


that  pertain),  to  descriptive  aids  which  offer  no  procedural  assistance  and  which 
are  principally  the  source  of  help  from  which  precise  actions  must  be  deduced. 

JPAs  are  usually  designed  with  particular  personnel  qualifications  in 
mind.  Fully  proceduralized  aids  are  more  appropriate  for  less  experienced 
personnel;  more  deductive  aids,  for  those  more  qualified.  Hybrid  aids  are 
designed  to  be  adaptive  to  the  level  of  expertise  of  their  users,  and  support 
performance  in  less  proceduralized  formats  and  more  deductive  formats  as  the 
user  increases  in  competence.  In  summary,  JPAs  are  means  of  extending  human 
capability  by  providing  a  means  of  storing  for  later  retrieval  information  related 
to  on-the-job  performance.  JPAs  may  reduce  or  even,  in  some  circumstances, 
eliminate  the  requirement  for  technical  training. 

Booher  found  that  satisfactory  aiding  techniques  exist  for  all 
nontroubleshooting  operation  and  maintenance  tasks  and  for  most  troubleshooting 
tasks  on  mechanical,  electrical,  and  electromechanical  systems.  However,  no 
aiding  techniques  were  satisfactory  for  complex  digital  electronic  systems. 
Troubleshooting  complex  digital  electronic  equipment,  therefore,  has  been 
identified  as  a  challenge  area  for  both  training  and  job  performance  aiding. 

Timely  and  accurate  production,  distribution,  and  maintenance  of  JPA 
systems  have  been  identified  as  major  weaknesses  of  JPA  systems.  The  major 
cost  factor  of  JPA  systems  is  front-end  analysis  and  the  amount  of  graphic 
enrichment  needed  to  increase  intelligibility. 

It  was  found  that  microform  and  audiovisual  systems  are  inferior  or 
equal  to  hard-copy  print  media.  For  most  job  environments,  simple  hard-copy 
print  media  JPAs  meet  the  information  aiding  needs  of  operators  and  technicians. 
Further  research  is  warranted  into  devices  that  can  increase  the  performance 
capabilities  of  the  user  beyond  that  possible  with  hard-copy  print. 

Interactive  JPA  devices  can  be  considered  in  a  class  by  themselves. 
Burkett  and  Kruse  (1977)  have  postulated  that  use  of  such  aids  may  cause  a 
reorganization  in  the  thinking  processes  of  technicians,  enhancing  the  users' 
capacity  to  develop  and  organize  thoughts,  define  necessary  steps,  and  develop 
the  most  efficient  procedures  for  problem  solutions.  Also,  these  aids,  through 
embedding  in  their  target  systems,  have  the  potential  for  modifying,  not  simply 
aiding,  task  performance.  Only  interactive  JPAs  are  identified  as  having  high 
potential  is  assisting  the  performance  of  cognitive  problem  solving  tasks. 

JPAs  have  many  of  the  same  logistical  and  quality  assurance  problems  as 
does  technical  training.  In  particular,  without  a  method  for  assessing  a  JPA's 
value  relative  to  other  JPAs  and  other  methods  of  developing  operational 
performance  capability,  objective  evaluation  of  the  utility  of  a  JPA  system  is  not 
possible. 

Booher  has  identified  several  priorities  for  future  JPA  R&D.  At  the 
systems  level,  the  effects  on  training,  the  personnel  system,  and  job  design 
variables  should  be  studied  while  holding  the  given  JPA  system  constant.  Work  on 
new  aiding  devices,  basic  comparison  performance  data,  and  hybrid  aids  is 


warranted.  R&D  on  readability  of  technical  instructions  is  recommended,  because 
the  standard  of  counting  syllables  per  word  and  translating  that  to  a  reading  grade 
level  is  not  an  adequate  method.  Finally,  R&D  into  systems  that  assist  in  the 
development  and  validation  of  JPAs  is  needed. 


Approaches 

Troubleshooting  digital  electronics  is  a  potential  priority  area  for 
increased  JPA  RdcD  emphasis.  Interactive  JPAs  were  identified  as  the  only 
appropriate  approach  to  this  area  (Booher,  1978).  Therefore,  the  application  of  AI 
technology  toward  the  development  of  interactive  computer-based  JPAs  for 
electronics  troubleshooting  is  suggested.  Such  interactive  JPAs  have  great 
potential  for  adapting  to  user  expertise  through  use  of  the  same  base  technology 
that  is  used  in  the  construction  of  adaptive  CAI.  Thus,  device-based  JPAs  in 
electronics  also  provide  an  ideal  test  bed  to  further  research  on  hybrid  JPAs  and 
adaptive  technical  orders  (TOs). 

NTEC  is  currently  investigating  the  development  of  a  Personal 
Electronic  Aid  for  Maintenance  (PEAM)  under  a  research  contract  with  Texas 
Instruments,  Inc.  The  primary  objective  of  PEAM  is  to  produce  performance 
specifications,  hardware,  and  software  for  a  PEAM  device. 

The  development  of  design  and  development  systems  is  an  important 
applications  goal  for  JPAs  and  for  courseware.  Such  systems  should  facilitate 
timely  JPA  revision.  Any  JPA  system  must  be  maintainable;  an  out-of-date  JPA 
is  no  aid  at  all. 

Readability  and  intelligibility  were  identified  as  areas  of  need. 
Currently  at  ONR,  a  number  of  related  projects  are  underway,  for  example: 
Comprehension  and  Analysis  of  Information  in  Text  and  Pictorial  Material  (Antos, 
Bourne,  Kintsch  &  Kozminsky,  1981);  Structual  Approaches  to  Procedural  Learning 
from  Text  and  Graphics  (Hirschfield  &  Bieger,  1981);  Understanding  and  Executing 
Instructions  (NR  154-461);  Topic  and  Theme  Identification  in  Prose  Comprehension 
(Kieras,  1981),  and  What  People  Know  About  Electronic  Devices:  A  Descriptive 
Study  (Kieras,  1982).  Much  of  this  work  has  cognitive  science  foundations  and  the 
results  of  this  work  should  be  incorporated  into  the  development  of  device-based 
JPAs,  as  well  as  in  the  development  of  design  and  development  aids  for  hard-copy 
print  media  JPAs. 

The  advent  of  expert  systems  has  opened  a  new  world  of  possibility  to 
job  performance  aiding.  Expert  systems  have  been  designed  which  have,  within 
their  boundary  of  expertise,  human  level  competence.  At  the  present  time, 
various  manufacturers  are  attempting  to  develop  intelligent  diagnostic  support 
aids. 


Traditional  distinctions  between  training,  job  performance  aiding,  and 
automated  test  equipment  are  beginning  to  blur  due  to  the  advent  of  the  expert 
system.  This  is  because  the  same  simulated  representations  of  task  and 
equipment  which  constitute  the  core  of  an  expert  system  are  fundamentally  useful 
whether  in  training,  helping  a  human  do  a  job,  or  in  doing  it  for  him  or  her. 
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Challenges 

Certain  information  necessary  for  job  performance  is  better  assigned  to 
the  "head"  via  training  than  to  the  "book"  via  JPAs  (Chalupsky  &  Knopf,  1967, 
Chenzoff,  1973).  Training  and  operational  performance  are  part  of  a  greater 
whole.  As  hybrid  or  adaptive  JPA  systems  become  available,  there  will  be  less 
and  less  reason  for  separating  training  and  aiding  in  the  initial  stages  of  enhancing 
job  knowledge  and  skills.  Teel  and  Chaney  (1965)  found  that  the  combination  of 
training  and  aiding  produced  almost  double  the  performance  improvement  of 
either  method  used  alone.  During  the  career  of  a  technician,  the  proper  mix  of 
training  and  aiding  varies.  Integrating  training,  assessment,  and  JPA  systems 
recognizes  this. 

Tradeoffs  between  JPAs,  training,  and  job  design  are  often  discussed  as 
if  each  of  these  technologies  were  equivalent.  They  are  highly  interrelated,  yet 
no  one  technology  is  a  direct  substitute  for  the  other.  For  example,  while  a  fixed, 
fully  proceduralized  JPA  yields  cost  effective  performance  for  entry-level 
technicians,  there  is  a  negative  cost  associated  with  not  training  these  people. 
This  shows  up  later  in  long-term  career  development  as  an  inadequacy  in  having 
and  being  able  to  generalize  complex  troubleshooting  skills. 

Job  performance  aiding  technology  and  technical  training  technology 
share  much  in  common,  in  particular  the  need  for  front-end  task  analysis  and  the 
need  for  formative  and  summative  evaluation.  Task  analyses  designed  to  cover 
the  requirements  of  all  three  issues  would  result  in  cost  savings  through  reduced 
duplication  of  effort  and  in  increased  effectiveness,  through  explicit  recognition 
of  the  close  relationship  between  training,  job  performance  aiding,  and  job  design. 


A  clear  challenge  is  to  develop  sophisticated  design  systems  that 
formally  take  these  issues  into  consideration  in  the  development  of  residence 
training,  on-the-job  training,  and  JPAs.  In  addition,  when  new  weapons  systems 
are  being  designed,  the  induced  effects  of  device  and  job  design  on  the  training 
and  aiding  requirements  can  be  explored  and  a  human-factored  design  may  be 
approached  in  a  systematic  fashion. 


Approaches 

A  kernel,  procedural  knowledge  base  can  be  developed  for  an  existing  or 
in-development  weapons  system  that  includes  the  device  representation,  a  task 
representation,  and  a  how-it-works  representation.  This  knowledge  base  can  be 
used  to  investigate  tradeoffs  between  job  performance  aiding,  residence  training 
and  OJT,  and  device  and  task  design.  Such  analyses  will  yield  empirically  based 
decisions  regarding  device  configuration,  job  design,  and  training  and  performance 
aiding  requirements.  It  is  then  possible  to  utilize  this  knowledge  base  in  concert 
with  appropriate  design  systems,  building  blocks,  and  tools  in  the  design, 


development,  and  delivery  of  adaptive  instruction  and  hybrid  JPAs.  As  seen  in 
Clancey's  experiments  with  MYCIN,  it  is  not  true  that  a  knowledge  base 
especially  suited  to  performance  would  be  necessarily  useful  for  instruction.  But 
that  should  not  be  a  problem  in  this  case  if  the  knowledge  base  is  designed  to 
support  a  basis  for  experienced  technicians  to  expand  their  skills,  whether  in 
training  or  on  the  job.  Systems  which  merely  require  the  input  of  symptoms  and 
respond  with  the  probable  fault  are  not  useful  in  this  regard.  The  Air  Force  goal 
is  to  build  a  basis  for  technicians  to  improve  their  performance  and  to  be  able  to 
transfer  their  skills  from  one  job  position  to  another,  contributing  to  a  long-term 
enhancement  of  the  level  of  human  expertise  on  the  job.  Therefore,  an  integrated 
adaptive  training/assessment/JPA  design,  development,  and  delivery  system 
addresses  the  ultimate  source  of  need  for  training  and  job  aiding  in  the  first 
place— a  shortage  of  expertise. 

If  this  approach  is  chosen,  it  is  critical  that  a  means  of  reducing  the 
labor  intensiveness  of  developing  and  using  the  central  data  base  be  developed. 
At  the  very  least,  tools  and  building  blocks  should  be  developed  to  facilitate  the 
construction  and  use  of  such  knowledge  bases.  Approaches  to  semi-automatic 
knowledge  extraction  have  been  mentioned  in  the  section  on  task  analysis.  It  is 
critical  that  the  central  data  base  be  able  to  accommodate  the  variety  of  views 
and  perspectives  and  have  appropriate  interfaces  with  each  user  community,  so 
that  each  can  effectively  interact  with  the  data  base— in  defining  it,  building  it, 
refining  it,  validating  it,  and  using  it.  Members  of  the  various  user  communities 
must  be  able  to  interact  with  the  data  base  individually,  and  directly,  otherwise 
its  maintainability  is  compromised.  A  system  that  cannot  be  supported  will  not  be 
used.  And  a  system  that  is  designed  to  be  used  by  many  people  with  different  yet 
closely  related  goals  will  not  be  supported  unless  these  people  have  direct  access 
to  it.  Direct  user  involvement  contributes  to  a  feeling  of  "ownership"  that  goes  a 
long  way  toward  assuring  that  the  system  will  be  accepted. 

With  a  community  involvement  with  a  common  data  base,  strong,  viable, 
systemic  mechanisms  must  be  in  place  to  ensure  that  the  data  base  and  its 
products  (training,  JPAs)  are  valid  and  remain  valid.  In  addition  to  traditional 
evaluation  procedures,  the  central  knowledge  base  can  be  used  to  develop 
measures  of  performance,  and  the  various  delivery  systems  can  be  used  to  collect 
performance  data,  with  which  to  validate  the  tradeoffs  chosen  and  the  utility  and 
effectiveness  of  the  training  and  3PA  system  which  have  been  designed. 

This  integrated  system  can  be  extended  to  incorporate  OJT  management 
procedures  and  the  tracking  of  individual  personnel  qualifications  at  a  level  of 
detail  useful  in  maintaining  an  ongoing  03T  program  for  a  particular  technician. 
Data  on  individual  qualifications  maintained  in  this  integrated  system  can  be 
interfaced  with  related  management  systems,  such  as  the  Integrated  Training 
System  (ITS)  and  the  Training  Decisions  System  (TDS).  These  interfaces  can  be 
developed  utilizing  AI  concepts  and  tools  which  may  contribute  to  the 
management  of  complex  systems. 
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CHAPTER  IV.  PRACTICAL  CONSIDERATIONS 


Early  in  its  history,  AI  held  forth  great  promise  of  near-term 
applications,  for  example,  in  language  translation  and  chess.  The  failure  of  this 
promise  to  materialize  was  a  great  blow  to  the  credibility  of  AI  research.  In  the 
interim,  as  discussed  in  Chapter  I,  AI  has  matured,  particularly  through  emphasis 
on  knowledge-based  systems.  Now,  once  again,  AI  is  holding  forth  promise. 
Numerous  private,  for-profit  ventures  seeking  to  capitalize  on  AI  applications 
have  recently  been  formed.  Public  awareness  of  AI  has  been  heightened  by  the 
Japanese  "fifth  generation  computer"  project.  DoD  has  taken  a  greater  interest 
in  near-term  applications,  an  interest  which  this  report  serves.  It  follows  that  AI 
procurement  commitments  must  be  carefully  targeted  for  success.  A  second 
failure  for  AI  on  its  second  rising  would  be  a  tremendous  blow  to  AI  and  to  those 
who  sought  to  apply  it. 

The  remainder  of  this  chapter  is  organized  around  the  following  practical 
issues:  available  hardware,  software  tools,  personnel  resources,  and  guidelines  for 
practical  efforts. 


Hardware 


Until  the  last  few  years,  almost  all  work  on  systems  of  the  sort  described 
here  has  been  performed  on  computers  whose  typical  million  dollar  price  tag  has 
been  a  barrier  to  both  RicD  and  practical  use.  Many  sites  have  tried  to  use 
existing  data  processing  facilities,  but  these  systems  did  not  offer  suitable 
languages  or  interactive  capabilities  and  were  ruled  out  immediately. 
Fortunately,  a  selection  of  much  less  expensive  equipment  now  exists  in  price 
ranges  acceptable  at  least  for  the  development  phase,  and  trends  indicate  that 
target  machines  for  delivering  this  type  of  system  at  low  cost  are  just  around  the 
corner.  For  example,  for  development  purposes,  high  performance  personal  LISP 
machines  (such  as  the  LM-2  and  3600  systems  available  from  Symbolics)  or  similar 
machines  could  be  used.  Other  comparable  systems  include  the  Xerox  "Dolphin" 
and  "Dorado,"  and  the  LMI  CADR.  The  Xerox  machines  support  INTERLISP  and 
are  compatible  with  the  "Dandelion,"  a  possible  delivery  vehicle.  The  Three 
Rivers  PERQ  has  many  similarities  to  the  systems  just  mentioned,  but  currently 
has  no  LISP  implementation. 

It  is  very  likely  that  M6800-based  systems  will  begin  to  be  available  with 
production-quality  LISP  environments,  once  the  68010  (virtual  memory)  version  of 
this  micro-processor  begins  to  be  widely  available.  The  following  systems  could 
serve  as  low  cost  target  systems  at  that  time:  Apollo  Domain,  Sun  Workstation, 
WICAT  150,  Fortune,  Corvus  Concept,  and  many  others.  Few  have  virtual 
memory  at  present.  (Large  virtual  memory  is  the  primary  architectural 
requirement  for  AI.)  In  some  specialized  situations,  the  IBM  Personal  Computer 
might  be  usable  as  an  AI  target  machine  if  equipped  with  Winchester  Disk  and 
sufficient  memory.  These  systems  at  present  have  no  production-quality  LISP.  (A 
number  of  LISP  implementations,  including  one  at  Utah  and  one  at  Yale,  are 
underway  at  present.) 


Software  Tools 


Creating  an  AI  application  means,  among  other  things,  creating 
software.  Software  languages,  environments,  and  higher  order  development 
systems  become  crucial  considerations  in  the  practical  development  of  an  AI 
application.  This  section  considers,  in  turn,  whether  LISP  is  essentially  a 
prerequisite  foundation  for  AI  programming  and  what  higher  order  development 
systems  are  available  and,  ultimately,  how  useful  these  can  be. 

Despite  the  benefits  of  widely  available  (and  allegedly  more  efficient) 
languages  such  as  FORTRAN  or  PASCAL,  or  the  structure  and  portability  of  Ada, 
the  current  technology  of  AI  employs  the  use  of  the  LISP  language.  Automatic 
storage  management,  the  elevation  of  functions  and  closures  to  "first  class 
citizen"  status,  the  concept  of  property  lists,  the  relationship  between  variables 
and  symbols,  and  the  central  role  of  an  interpreter  (rather  than  a  compiler)  are 
features  of  the  language  arguing  for  its  use  in  this  type  of  application.  Other  AI 
languages  have  been  developed  (Barr  &  Feigenbaum,  1982),  but  LISP  in  its  various 
dialects  predominates  as  the  base  language. 

Attributes  that  characterize  Al-supportive  programming  languages  in 
general  include  the  following: 

1.  Focus  on  symbol  manipulation  and  list  processing. 

2.  Support  of  representations,  which  change  dynamically. 

3.  Support  of  flexible  control  by  pattern  matching  rather 

than  by  procedure  calls. 

4.  Supportive  programming  environment,  including 

a.  An  interactive  (interpreted)  language. 

b.  A  good  editor  (program  construct  oriented,  not  text 
oriented). 

c.  Debugging  facilities  (traces,  breaks). 

d.  Standard  systems  input/output  functions. 


High  Order  Development  Systems 

There  are  numerous  AI  "building  block"  systems  available,  some  from 
universities  without  support,  and  some  commercially.  An  expert  systems 
workshop  (funded  by  the  National  Science  Foundation  and  the  Defense  Advanced 
Research  Projects  Agency)  was  recently  held,  resulting  in  two  important  products. 
One  (Waterman  &  Hayes-Roth,  1982)  is  a  report  describing  an  investigation  into 
the  comparative  merits  of  eight  very  high  level  programming  languages  designed 
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for  expert  systems:  EMYCIN,  KAS,  EXPERT,  OPS5,  ROSIE,  RLL,  HEARSAY-III, 
and  AGE. 


The  other  product  is  a  book  describing  how  to  build  expert  systems 
(Hayes-Roth,  1983).  This  comparative  study  describes  available  systems,  their 
sources,  their  comparative  merits,  and  lessons  learned  from  a  comparative 
analysis  of  these  systems,  in  particular,  insights  regarding  choosing  a  tool 
appropriate  for  the  problem  and  designing  a  tool  for  expert  system  building. 

Quoting  the  comparative  study  (Waterman  &  Hayes-Roth,  1982,  p.  47), 
maxims  suggested  for  choosing  an  appropriate  tool  are: 

1.  Do  not  pick  a  tool  with  more  generality  than  you  need. 

2.  Test  your  tool  early  by  building  a  small  prototype 
system. 

3.  Choose  a  tool  that  is  maintained  by  the  developer. 

4.  Use  the  problem  characteristics  to  determine  the  tool 
features  needed. 

Various  tools  exist  for  knowledge  representation,  in  particular,  KRL 
(Bobrow  &  Winograd,  1977)  and  SRL  (Wright  &  Fox,  1982). 

For  specific  applications,  one  should  consider  developing  tools  and 
building  blocks  from  scratch.  An  expert,  using  a  LISP  machine,  could  develop  an 
inference/retrieval  package  in  a  week  or  two  of  intense  effort.  The  resulting 
package  is  apt  to  be  more  suitable  for  the  specific  application  than  might  be  the 
general  packages  now  available.  However,  in  the  long-term,  the  use  of  such 
building  blocks  as  components  in  larger  systems  will  be  essential. 


Personnel 


People  with  formal  training  or  practical  experience  with  AI  are 
extremely  rare.  The  recent  interest  in  AI  by  the  private  sector  has  created  an 
enormous  demand  for  AI  experts,  creating,  for  the  experts,  a  sellers'  market.  This 
means  self-aware  AI  experts  expect  high  salaries,  interesting  work  assignments, 
software  and  hardware  environments  they  are  familiar  with  and  like  to  use,  and 
evidence  of  management  commitment  to  their  work.  Because  of  this,  other 
practical  considerations,  such  as  choice  of  hardware  or  software  are  often  forced 
by  personnel  choices.  This  is  especially  germane  to  the  LISP  issue:  does  one  need 
to  do  AI  in  LISP?  The  question  is  moot  if  it  is  true  that  most  AI  experts  like  to 
and  are  accustomed  to  programming  in  LISP  and  may  not  be  interested  in  working 
on  a  project  that  does  not  use  LISP  or  one  of  the  higher  level  tools  built  upon 
LISP. 


The  common  advice  from  people  involved  in  AI  projects  is  to  recruit 
people  with  proven  AI  expertise  and  not  simply  to  declare  someone  in  the  data 
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processing  department  as  the  AI  expert.  At  the  same  time,  due  to  the  shortage  of 
expertise,  an  organization  should  make  every  possible  effort  to  develop  its  own 
expertise  internally.  Martins  (1982)  suggests  an  AI  effort  be  staffed  as  follows: 
recruit  talent  to  fill  the  main  initial  technology  requirements;  use  the  best  of 
existing  staff  for  technology  support.  With  this  strategy,  the  following  dynamics 
will  result:  technically  gifted  but  probably  managerially  unsophisticated  AI 
recruits  will  mature  in  leadership  qualities  as  the  existing  staff,  already  attuned 
to  organizational  realities,  acquire  technical  competence  in  A I. 

The  personnel  "crunch"  is  widely  recognized  by  managers  of  DoD 
applications  programs.  A  solution  to  the  shortage  is  addressed  most  directly  in  a 
training  and  education  program  proposed  to  the  Joint  Logistics  Commanders.  This 
program  would  provide  funding  for  increased  higher  education  opportunities  in  AI 
and  hopefully  help  to  alleviate  the  undersupply  of  trained  personnel  in  this  field. 

As  mentioned,  AI  is  becoming  a  commercial  venture;  so,  to  a  certain 
extent,  expertise  can  be  directly  contracted  for  with  firms  active  in  this  area. 
Reports  on  those  firms  have  been  made  in  the  popular  press  (cover  story  in 
Business  Week,  March  8,  1982;  a  major  article  in  Time,  May  17,  1982).  Of  course, 
in  addition  to  sites  of  industrial  competence,  there  are  the  basic  university, 
nonprofit  and  governmental  centers  of  activity.  These  too  may  serve  as  sources 
of  expertise  and  potential  sources  for  recruitment. 


Guidelines  for  Practical  Efforts 


Today's  applications  promise  of  AI  is  real,  but  past  over  zealousness  has 
taught  a  lesson.  In  any  applications  effort  contemplated  today,  the  following 
principles  of  practical  and  achievable  project  management  must  be  followed. 

The  scope  of  the  applications  effort  must  be  constrained  to  highly 
specialized  solutions  to  narrowly  defined  problems.  From  Hayes-Roth  (Gevarter, 
1982,  p.  2):  "the  problem  should  be  nontrivial  but  tractable,  with  promising 
avenues  for  incremental  expansion."  Objective,  definable  interim  milestones 
should  be  set.  Hayes-Roth  (1982,  p.  410-411)  lists  some  situations  that  instigate 
knowledge  engineering  initiatives  and  some  kinds  of  product  innovations  now 
possible: 

Situations  that  Instigate  Knowledge  Engineering  Initiatives 

1.  The  organization  requires  too  many  skilled  people  to 
recruit  or  retain. 

2.  Problems  arise  that  require  almost  innumerable 
possibilities  to  be  considered. 

3.  Job  excellence  requires  a  scope  of  knowledge  exceeding 
reasonable  demands  on  human  training  and  continuing 
education. 
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4.  Problem  solving  requires  several  people  because  no 
single  person  has  the  needed  expertise. 

5.  The  company's  inability  to  apply  its  existing  knowledge 
effectively  now  causes  management  to  work  around 
basic  problems. 

Kinds  of  Product  Innovations  Now  Possible 

1.  Active  instruments  that  solve  problems  by  producing 
answers  instead  of  data. 

2.  Systems  that  explain  how  they  work  and  how  to  use 
them. 

3.  Corporate  memories. 

4.  Reasoning  aids  and  thinking  prosthetics. 

5.  Accident-proof  machines. 

6.  Automated  sellers. 

7.  Hypothesis  and  expectation  management  systems. 

Realistic  project  cost  and  personnel  effort  estimates  are  crucial  inputs 
to  successful  applications.  These  costs  depend  on  the  availability  of  tools. 
Hayes-Roth  (1982,  p.  412)  cites  as  the  resources  required  for  a  typical  knowledge 
engineering  project  (about  500  knowledge  "chunks"  in  the  rule  base),  utilizing 
existing  tools,  to  be  about  $300,000  to  $1,500,000  with  2  to  5  personnel  years. 
Included  in  this  estimate  is  design,  development,  knowledge  engineers,  computing 
and  overhead. 

Another  analysis,  based  on  project  costs  of  exemplary  AI  projects  over 
the  past  15  years  led  Davis  (1982,  p.  10)  to  the  following  conclusion: 

Even  for  the  best-understood  problems,  experienced 

researchers  using  the  best-understood  technologies  still 

require  at  least  five  man-years  to  develop  a  system  that 

begins  to  be  robust. 

The  stages  of  development  of  an  expert  system  include  (Davis,  1982) 
system  design,  system  development  (conference  paper  level),  formal  evaluation  of 
performance,  formal  evaluation  of  acceptance,  extended  use  in  prototype 
environment,  development  of  maintenance  plans,  and  system  release.  Davis  notes 
that,  to  date,  no  current  system  has  completed  all  these  stages. 


Case  Study:  Practical  Guidelines  for  AI  Applications  Efforts 

In  order  to  add  substance  to  the  observations  regarding  development 
guidelines,  this  paper  includes  as  a  case  study  the  considerations  paid  those  issues 
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by  one  of  the  first  commercial  attempts  to  capitalize  on  AI  technology, 
Computer*  Thought  Corporation's  Ada  Tutor. 

This  section  describes  the  current  Computer*  Thought  product 
development  effort  as  a  case  study  in  practical  AI-CAI  applications  projects.  The 
reader  should  be  aware  that  the  project  is  not  yet  complete,  and  so  conclusions 
must  be  treated  as  tentative  and  subject  to  modification.  The  material  in  the 
remainder  of  this  chapter  was  provided  by  Dr.  Mark  L.  Miller  of 
Computer*  Thought. 

How  was  the  scope  of  the  practical  applications  effort  determined  and 
specified? 

The  founders  of  Computer*  Thought  had  been  involved,  over  a  number  of 
years,  in  developing  and  working  with  prototype  systems  for  teaching  computer 
programming.  It  was  desired  to  apply  this  technology  to  some  need  that  would  be 
sufficient  to  warrant  the  cost  of  using  systems  in  the  Vax  or  LISP  machine  class 
for  initial  deliveries,  on  the  assumption  that  personal  computers  with  this 
performance  will  be  commonplace  in  the  mid  to  late  eighties.  With  DoD  software 
expenditures  in  the  billions  of  dollars  per  year,  and  many  embedded  systems 
contracts  involving  millions  of  dollars,  the  Ada  training  market  provided  an  ideal 
opportunity.  Computer* Thought  is  developing  a  generic  AI-CAI  technology  for 
teaching  computer  programming,  using  Ada  as  the  initial  vehicle.  As  hardware 
costs  continue  to  decline,  programming  tutors  for  new,  potentially  mass-market 
languages  (e.g.,  LOGO  or  SMALLTALK)  will  be  natural  follow-ons  from  this 
technology).  The  project  scope  (in  terms  of  both  curriculum  and  system  features) 
was  narrowed  until  a  small  team  (growing  to  a  dozen  technical  staff)  could 
complete  an  initial  product  implementation  within  1  year. 

What  effect  did  the  following  conditions  have  on  the  effort? 

1.  The  specificity  of  the  problem. 

Problem  specificity  is  the  key  ingredient  to  success  in  building  this 
type  of  system  at  the  present  time.  As  the  project  has  progressed, 
the  system  designers  have  continued  to  revise  initial  requirements 
to  increase  specificity.  There  are  many  topics  one  might  include  in 
an  Ada  curriculum,  for  example,  and  many  features  one  might 
include  in  such  a  system.  The  question,  "will  this  have  a  first-order 
effect  on  overall  effectiveness?"  has  helped  to  focus  system  goals. 

2.  Existence  of  prior  related  R&D  and  prototypes. 

The  Ada*Tutor  project  was  specifically  chosen  because  the  company 
founders  had  experience  with  several  such  R&D  prototypes.  In 
particular,  the  BASIC  Instructional  Program  (BIP),  the  Spade 
system,  ACSES,  and  TURTLE  were  previous  prototypes  which  guided 
the  design.  Also,  existing  experimental  literature  on  help  systems 
for  learning  programming  was  useful. 
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3.  Project  resource  expenditures  as  a  function  of  time  or  phase. 

Computer* Thought's  initial  capitalization  has  been  almost  entirely 
devoted  to  the  Ada*Tutor  project.  Staffing  has  grown  from  the  2 
founders  to  11  at  present  and  is  planned  to  grow  to  about  20  by 
summer  of  1983.  Approximately  80  percent  of  staff  members  have 
primarily  technical  functions. 

4.  Personnel  qualifications,  background,  mix. 

Unfortunately,  such  projects  currently  require  a  rare  type  of  talent 
that  is  in  great  demand  in  industry.  Very  few  universities  offer  any 
formal  training  in  this  area.  A  team  of  4  to  12  technical  people, 
organized  around  a  single  technical  leader,  seems  most  effective.  A 
certain  amount  of  administrative  assistance,  as  well  as  junior  staff 
(e.g.,  AI  graduate  students)  can  multiply  the  productivity  of  key 
personnel  considerably.  It  takes  several  years  of  intense  study  to 
achieve  expertise  in  AI/LISP  programming;  hence  reassigning 
conventional  computer  science  personnel  to  this  task,  without 
formally  trained  outside  expertise,  is  not  recommended. 

5.  Hardware/software  tools. 

The  level  of  ambition  of  the  current  Ada*Tutor  could  not  have  been 
achieved  within  the  available  resources  and  time  were  it  not  for 
high  performance  personal  LISP  machines  and  the  programming 
environment  they  provide.  Graphics  features  such  as  "windows,"  and 
interaction  features  such  as  menu  selection  using  a  "mouse,"  have 
made  it  possible  to  rapidly  prototype,  demonstrate,  and  re-evaluate 
ideas  within  a  few  weeks.  AI  programming  tools,  such  as  the 
ZMACS  LISP-specific  screen  editor  and  the  ability  to  rapidly  locate 
callers/users  of  a  symbol  within  a  large  system,  have  been  essential 
factors  in  project  progress. 

What  are  the  stages  in  such  a  project? 

AI  product  development  projects  are  really  too  new  to  reach  any  sort  of 
generalization  at  present.  It  seems  clear  that  traditional  software  project  life- 
cycle  models  do  not  carry  over  well  into  the  AI  and  AI-CAI  arenas.  Current 
experience  seems  to  be  settling  on  four  more  or  less  distinct  phases:  (a) 
requirements  analysis  and  initial  design,  (b)  implementation  of  an  initial 
feasibility  demonstration,  (c)  prototype  development  and  initial  testing,  and  (d) 
beta  testing  and  final  revision.  There  remains  a  danger  that  the  level  of 
robustness  required  for  the  final  product  (versus  a  prototype)  will  be  more 
difficult  to  achieve  than  was  originally  contemplated. 


CHAPTER  V.  CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 


Training,  performance  measurement,  and  job  performance  aiding  are 
neither  separate  nor  independent  Air  Force  activities.  Training  (both  in-residence 
and  on-the-job)  and  job  performance  aiding  are  related  because  they  are 
alternative,  complementary  approaches  to  the  problem  of  supporting  performance 
capability  in  career  specialties.  Performance  measurement  is  related  to  both 
training  and  job  performance  aiding  because  it  is  through  performance 
measurement  that  training  and  JPAs  are  validated,  evaluated  and  improved.  The 
close  relationship  among  these  three  disciplines  is  not  limited  to  their 
interdependence.  They  are  also  related  because  scientific  approaches  to  each  of 
these  three  disciplines  are  based  on  task  analysis. 

The  technology  and  techniques  of  AI  are  based  on  development  of 
specific  knowledge  representations  and  sets  of  procedures  that  act  on  those 
representations.  The  performance  capability  of  intelligent  systems  is  based  on 
the  representation  and  utilization  of  large  amounts  of  knowledge.  Job  task 
analysis  serves  as  the  means  of  providing  a  representation  that  can,  in  turn,  be 
processed  by  training,  performance  measurement,  and  JPA  systems.  Therefore, 
the  fundamental  relatedness  of  this  knowledge  may  be  capitalized  on  in  the 
development  of  intelligent  systems  to  serve  these  disciplines. 

From  a  review  of  the  challenges  facing  each  of  these  disciplines 
separately,  it  appears  that  the  development  of  adequate  job  task  analyses  is  a 
basic  problem.  From  a  review  of  AI  technology,  it  appears  that  AI  provides  a 
unique  opportunity  to  develop  and  validate  representations  that  can  serve  as  task 
analyses.  These  representations  are  machine-readable  and  can  be  used  with  the 
help  of  AI  techniques  to  simulate  intelligent  behavior,  for  example:  to  perform 
the  job  task  of  interest,  or  to  develop  or  provide  instruction  in  the  job  task  of 
interest.  AI  (a)  provides  a  mechanism  for  developing  valid  task  analyses  that  help 
solve  one  of  the  major  problems  facing  all  three  of  the  potential  application  areas 
of  interest,  (b)  provides  a  basis  for  integrating  the  three  applications  areas  into  a 
formal  integrated  system,  and  (c)  does  so  computationally,  providing  the 
foundation  for  an  automated,  computer-based  system  to  serve  as  an  aid  in 
increasing  productivity  and  effectiveness  in  training,  performance  measurement, 
and  job  performance  aiding. 

The  principal  recommendation  to  be  derived  from  these  observations  is 
that  training,  performance  measurement,  and  job  performance  aiding  should  be 
treated  as  an  integrated  system,  and  that  an  integrated  system  is  the  natural  way 
to  approach  AI  applications  in  each  of  these  areas.  Since  AI  applications  are 
computer-based;  a  computer-based  system  is  the  required  medium  in  which  to 
meet  the  challenges. 

No  integrated  computer-based  system  for  training,  measurement,  and  job 
performance  aiding  exists.  There  is,  therefore,  a  great  opportunity  here  to  design 


and  build  a  computer-based  system,  capitalizing  on  scientific  approaches  to 
building  a  common  foundation  of  task  analysis.  Whether  such  a  system  should  be 
non-AI-based  or  Al-based  may  be  questioned.  However,  it  is  relevant  to  note 
that  AI  itself  is  founded  on  formal  representations  and  the  procedures  acting  upon 
them,  and  therefore  offers  a  set  of  tools  and  techniques  that  are  almost  an  exact 
match  for  the  requirements  of  integrating  the  three  applications  areas. 

Traditional  approaches  in  each  of  these  areas  have  weaknesses  in  the 
area  of  knowledge  representation.  AI  offers  approaches  to  meeting  the  challenges 
faced  by  each  discipline  separately  and  offers  a  way  of  integrating  these 
disciplines  in  a  formal,  explicit  fashion. 


Recommendations 


Specific  recommendations  for  the  development  of  an  R<5cD  program  in 
applications  of  A!  technology  to  training,  performance  assessment,  and  job 
performance  aiding  are  presented  below.  The  approach  taken  is  to  structure  such 
a  program  around  an  integrated  technical  training,  performance  measurement, 
and  job  performance  aiding  design,  development,  delivery,  and  evaluation  system. 
The  details  of  specific  approaches  within  each  of  the  areas  of  training, 
measurement,  and  job  performance  aiding  are  presented  in  the  Approaches 
sections  in  the  main  body  of  this  report. 


Recommendation  1 

A  single,  integrated,  intelligent,  computer-based  system  should  be 
developed  having  application  to  three  facets:  training,  performance 

measurement,  and  job  performance  aiding. 


Recommendation  2 

The  same  system  should  support  in  an  integrated  fashion  the  design, 
development,  delivery,  and  evaluation  of  each  facet. 


Recommendation  3 

The  integrated  computer-based  system  should  support  and  encourage 
direct  interaction  with  instructional  developers,  subject  matter  experts,  course 
personnel,  on-the-job  supervisory  personnel,  evaluators,  and  the  trainee/specialist. 


Recommendation  4 

A  master  plan  specifying  the  basic  functions  of  the  computer-based 
system  should  be  developed  and  used  to  prioritize  its  own  implementation  so  that 


the  task  analysis  representations  are  developed  first  and  that  each  project  effort 
can  be  integrated  with  the  larger  system. 


Recommendation  5 

As  early  as  possible,  the  system  should  be  configured  as  a  program 
demonstration. 


Recommendation  6 

A  target  career  specialty  should  be  selected  through  careful  front-end 
analysis  with  the  objective  of  identifying  a  job  area,  first,  that  is  of  critical 
importance  in  the  Air  Force  mission,  second,  for  which  there  is  promise  or  need  of 
improved  productivity  and  effectiveness  or  of  reduced  cost  of  operations,  and 
third,  that  is  amenable  to  the  detailed  analyses  required  to  develop  device  and 
task  performance  models.  Avionics  maintenance  is  identified  as  a  general  area 
meeting  the  above  criteria. 


Recommendation  7 


The  central  procedural  and  declarative  knowledge  bases  should  include 
(a)  a  procedural  representation  of  the  device(s)  from  the  perspectives  of  both 
operations  and  maintenance,  (b)  a  procedural  representation  of  the  tasks,  both 
routine  operations  and  maintenance,  (c)  a  suitably  detailed  "how-it-works" 
hierarchy  regarding  the  device's  functionality  and  structure,  (d)  an  explicit  model 
of  the  interrelationships  between  training,  measurement,  and  job  performance 
support  and  evaluation  for  the  career  area  chosen,  and  (e)  an  explicit  body  of 
procedural  knowledge  relating  to  curriculum  sequencing  and  pedagogical 
interaction. 


Recommendation  8 

The  system  should  be  developed  in  LISP  or  alternatively,  in  Ada 
(assuming  that  comparability  can  be  established  between  LISP  and  Ada).  All 
aspects  of  the  system  should  share  the  same  computing  environment;  hence,  the 
delivery  modality  of  the  system  should  also  be  in  LISP. 


Recommendation  9 

The  system  should  utilize  start-of-the-art  graphics,  including  high 
resolution  graphics  and  graphic  overlays  on  video  disc  images. 
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Recommendation  10 


The  system's  terminals  or  stations  should  be  networked.  Local 
processing  should  be  supported,  but  it  is  necessary  for  data,  in  particular  the 
procedural  and  declarative  knowledge  bases,  to  be  stored  centrally.  These 
knowledge  bases  will  be  undergoing  continual  development  and  refinement 
throughout  the  design,  development,  delivery  and  evaluation  process. 

Recommendation  11 


System  hardware  should  be  state-of-the-art  LISP  machines  or  carefully 
chosen  alternatives.  The  long-term  implementation  strategy  should  rely  on  the 
fact  that  delivery  versions  of  these  machines  will  be  on  the  market  in  3  to  5 
years. 
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APPENDIX  A 


CURRENT  RELEVANT  DoD  RESEARCH  PROJECTS--DETAILED  LISTING 


The  information  in  this  section  is  divided  into  the  following  fields: 

Project  Title 

sponsoring  agency  performing  organization  project  duration 

project  monitor  project  contact  level  of  funding 

telephone  number  of  monitor  telephone  number  of  contact 

Reference  to  material  that  will  provide  additional  information 


For  each  title,  the  maximum  amount  of  information  available  has  been 
included.  The  information  is  presented  alphabetically  by  title;  Table  I  may  be  one 
useful  index  for  locating  project  titles  in  Appendix  A,  as  it  is  arranged  by 
sponsoring  agency. 

Abbreviations  of  the  sponsoring  agencies  are  as  follows: 


AFHRL 

AFOSR 

ARI 

NAVAIRENGCEN 

NPRDC 

NTEC 

ONR 

TAEG 


Air  Force  Human  Resources  Laboratory 

Air  Force  Office  of  Scientific  Research 

Army  Research  Institute 

Naval  Air  Engineering  Center 

Navy  Personnel  Research  and  Development  Center 

Naval  Training  Equipment  Center 

Office  of  Naval  Research 

Training  Analysis  Evaluation  Group  (Navy) 
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Adaptive  Computerized  Training  System  (ACTS) 


ARI  Perceptronics,  Inc. 

B.W.  Knerr  A.  Freedy  $521,000 

(202) 274-8872  (213) 884-7470 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DT1C,  1982. 


Advanced  Computer-Based  Training  for  Propulsion  and  Problem  Solving: 
STEAMER 

NPRDC  Bolt  Beranek  <5c  Newman,  Inc.  March  81 -March  84 

3.D.  Hollan,  Or.  A.  Stevens  $2,198,543 

(714)  225-7121  (617)  492-3656 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


Advanced  Instructional  Development:  Intelligent  Maintenance  System 

ARI  Sep  83-Jan  86 

3.  Psotka 
(202) 274-5540 


Analysis  of  Acquiring  Strategic  Knowledge  in  Problem  Solving 

ONR  (667-459)  University  of  Pittsburgh 

J.G.  Greeno  &  M.  Riley 
(412) 624-4141 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Artifical  Intelligence  Analyses 

NTEC  University  of  Central  Florida  Oun  81-Sept  82 

E.  Ricard  D.  Abbott  $201,992 

(305)  646-5130  (305)  275-221 6 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


Artificial  Intelligence-Based  Maintenance  Tutor 


ARI 

J.  Psotka 
(202)  274-5540 


(project  in  preparation) 


Artificial  Intelligence  in  Maintenance  Diagnostics  and  Information  Systems 


AFHRL 


Systems  Exploration,  Inc. 


Authoring  Instructional  Materials  (AIM) 


NPRDC 
J.D.  Ford 
(714)  225-7121 


NPRDC 
W.H.  Wulfeck 
(714)  225-7121 


Oct  81-Sept  86 


Defense  Technical  Information  Center.  Work  Unit  Summaries.  AleAndria,  VA: 
DTIC,  1982. 


Automated  Performance  Measurement  System  C-5  Aircraft 

AFHRL  (Williams  AFB) 

R.  Hughes 
(602)  988-6561 

Air  Force  Human  Resources  Laboratory.  AFHRL  Annual  Report:  FY  81.  Brooks 
AFB,  TX:  AFHRL,  1982. 


Automatic  Knowledge  Acquisition  Techniques  for  Expert  Systems 


NTEC 
R.  Ahlers 
(305)  646-5130 


Percept  ronics,  Inc. 
(213)884-7470  , 


$80,000 


Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


CASDAT  Enhancement 

NTEC 
R.  Bird 

(305)  646-4701 


Rowland  and  Company 
N.  Marcue 
(305)  894-7086 


Sept  81-Jul  82 
$93,831 
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Class  "A"  Electronic  Equipment  Maintenance  Training  System 


NPRDC 
V.M.  Malec 
(714)  225-7121 

Navy  Personnel  Research  and  Development  Center.  NPRDC  FY-82  Programs. 
San  Diego,  CA:  NPRDC,  n.d. 


Cognitive  and  Instructional  Factors  in  the  Acquisition  and  Maintenance  of  Skills 

ONR  (157-430)  University  of  Pittsburgh  Jan  79-Sept  82 

M.3.  Farr  R.  Glaser,  A.  Lesgold,  $1,298,950 

J.  Greeno,  M.  Chi,  &  M.  Resnick 
(202)  696-4504  (412)  624-4895 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


Cognitive  Processes  in  Planning  and  Control 

ONR  (667-411)  Rand  Corporation 

B.  Hayes-Roth 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1981  Programs. 
Arlington,  VAs  Office  of  Naval  Research,  December  1981. 


Comprehension  and  Analysis  of  Information  in  Text  and  Pictorial  Materials 

ONR  (667-422)  University  of  Colorado 

L.E.  Bourne,  Jr.,  W.  Kintsch,  & 

P.  Baggett 
(303)  492-0111 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Computational  Models  of  Problem-Solving  Skill  Acquisition 

ONR  (667-477)  Xerox  Palo  Aito  Research  Jan  82-Dec  82 

Center 

H.  Halff  J.S.  Brown  $98,730 

(202)  696-4327  (415)  494-4341 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


Computer  Aided  Authoring  System 


TAEG  University  of  Central  Florida 

Hosni  $39,000 

(303) 275-9101 


Computer-Based  Maintenance  Aids  for  Technicians 

AFHRL  (Wright-Patterson  AFB) 

D.L.  Thomas 
(513)  255-3771 

Air  Force  Human  Resources  Laboratory.  AFHRL  Annual  Report:  FY  81.  Brooks 
AFB,  TXs  AFHRL,  1982. 


Computer-Based  Modeling  and  Diagnostic  Testing  of  Problem-Solving  Skills 

ONR  (154-508)  Carnegie- Mellon  University  Nov  82-Nov  84 

H.  Haiff  P.  Langley  $203,425 

(202)  696-4327  (412)  578-3598 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1982  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1982. 


Computer-Based  Propram  Consultant  for  Introductory  Programming 

ONR/ARI  Yale  University  Jun  83-Sep  84 

R.A.  Wisher  E.  Soloway 

(202)  274-5779 

Personnel  Training  and  Research  Group  of  the  Office  of  Naval  Research.  News  & 
Notes.  1982,  2(4),  2-5. 


Computer-Based  Techniques  for  Training  Procedural  Skills 

NPRDC 
E.L.  Hutchins 
(714) 225-7121 

Navy  Personnel  Research  and  Development  Center.  NPRDC  FY-82  Programs. 
San  Diego,  CA:  NPRDC,  n.d. 


Computer-Based  Training  of  Mission  Planning  and  Control  Skills 

ONR  (154-486)  Perceptronics,  Inc.  Feb  82-Feb  85 

H.M.  Halff  P.W.  Thorndyke  $225,000 

(202)  696-4327  (213)  884-7470 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982.  T 


Computer-Based  Tutoring  for  Diagnostic  Skill  Acquisition 

ONR/ARI  Stanford  University  Mar  79-Sep  85 

3.  Psotka  B.  Buchanan  &  W.  Clancey  $1,110,447 

(202)  274-5540  (415)  497-1997 

Personnel  and  Training  Research  Group  of  the  Office  of  Naval  Research.  News  & 
Notes.  1982,  2(4),  2-5. 


Development  of  Draft  Military  Specifications  for  Maintenance  Task  Analysis  and 
Logic  Tree  Troubleshooting  Aids 

AFHRL  (Wright-Patterson  AFB) 

E.G.  McFall 
(513)  255-5910 

Air  Force  Human  Resources  Laboratory.  AFRHL  Annual  Report:  FY  81.  Brooks 
AFB,  TX:  AFHRL,  1982. 


Evaluation  of  Individual  Performance  in  Mechanical  Specialties 

AFHRL  (Brooks  AFB) 

S.  Lipscomb 
(512)  536-3551 

Air  Force  Human  Resources  Laboratory.  AFRHL  Annual  Report:  FY  81.  Brooks 
AFB,  TX:  AFHRL,  1982. 


General  Patterns  of  Scientific  Inference 

ONR/ARI  Carnegie-Mellon  University 

3.  Psotka  3.  Larkin  ic  3.  Carboneli 

(202)  274-5540  (412)  578-2000 


Personnel  and  Training  Research  Group  of  the  Office  of  Naval  Research.  News  A 
Notes.  1982,  2(4),  2-5. 


Human  Understanding  of  Complex  Systems 

ONR  (667-428)  Bolt  Beranek  A  Newman,  Inc.  Mar  79-Feb  82 

M.3.  Farr  A.M.  Collins  A  A.  Stevens  $292,531 

(202)  696-4504  (617)  497-3377 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


Intelligent  Automated  Tutors  for  Instruction  in  Planning  and  Computer 
Programming 

ONR  (154-461)  Texas  Instruments,  Inc.  (exp  date)  Sept  81 

H.M.  Halff  M.  Miller  $99,500 

(202)  696-4327  (214)  980-6081 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VAs 
DTIC,  1982. 


Intelligent  Computer-Assisted  Instruction:  An  Interactive  Reading  Tutor 

ONR/ARI  Yale  University  3un  83-Sep  84 

R.A.  Wisher  R.  Schank  A  3.  Black 

(202)  274-5779  (203)  436-4771 

Personnel  and  Training  Research  Group  of  the  Office  of  Naval  Research.  News  A 
Notes,  1982,  2(4),  2-5. 


Intelligent  Computer-Based  Instruction  for  Simulation  Training  of  Troubleshooting 
Skills 

ONR/ARI  Bolt  Beranek  A  Newman  May  83-Sep  84 

R.A.  Wisher  W.  Fuerzeig  &  3.  Frederiksen 

(202)  274-5779  (617)  497-3448 

Personnel  and  Training  Research  Coup  of  the  Office  of  Naval  Research.  News  A 
Notes.  1982,  2(4),  2-5. 


Intelligent  Instructional  Methods  for  Teaching  Procedural  Skills 

ONR  (154-493)  Bolt  Beranek  A  Newman,  Inc. 

W.  Fuerzeig  A  3.  Frederiksen 
(617)  497-3448 

Office  of  Naval  Research.  Psychological  Sciences  Division,  1982  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1982. 
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Modeling  Student  Acquisition  of  Problem-Solving  Skills 

ONR  (154-444)  Rutgers  University  Sept  79-Aug  81 

H.M.  Halff  R.L.  Smith  $129,992 

(202)  696-4327  (201)  932-3623 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VAs 
OTIC,  1982. 


Personal  Electronic  Aid  for  Maintenance  (PEAM) 

NTEC  Texas  Instruments,  Inc. 

B.  Rizzo  $100,000 

(305) 646-5130 


Plan  for  Application  of  Artificial  Intelligence  Technology  to  Technical  Training, 
Performance  Measurement,  and  Job  Performance  Aiding 

AFHRL  Denver  Research  Institute  Jan  83 -Sep  83 

G.  Walker  3.  Richardson  $125,000 

(303)  370-3391  (303)  753-2087 


Principles  of  Intelligent  Computer-Based  Instruction  for  Cognitive  Skills 

ONR/ARI  University  of  Pittsburgh  Mar  83-Sep  85 

J.  Psotka  J.  Greeno  &  J.  Brown 

(202)  274-5540  (412)  624-4141 

Personnel  and  Training  Research  Group  of  the  Office  of  Naval  Research.  News  & 
Notes.  1982,  2(4),  2-5. 


Procedural-Net  Theories  of  Human  Planning  and  Control 

ONR  (667-414)  Science  Applications,  Inc.  Jan  78-Jun  81 

M.J.  Farr  M.  Atwood  $232,486 

(202)  696-4504  (303)  773-6900 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


The  Role  of  Prior  Knowledge  In  Operating  Devices 

ONR  (667-473)  University  of  Arizona 

D.  Kieras 
(602) 626-2751 

Office  of  Naval  Research.  Psychological  Sciences  Division,  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 
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Simulation  Models  for  Procedural  Learning  and  Expert  Memory 

ONR  (157-465)  Carnegie-Mellon  University  Oct  80-Nov  82 

M.3.  Farr  3.R.  Anderson  $333,432 

(202)  696-4504  (4 1 2)  578-2781 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria.  VA: 
DTIC,  1982. 


Structural  Approaches  to  Procedural  Learning  from  Technical  Text  and  Graphics 

ONR  (667-452)  Cornell  University 

M.D.  Clock 
(607)  256-1000 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Studies  of  Skilled  Human  Performance 

ONR  (667-437)  University  of  California, 

San  Diego 

D.A.  Norman  &  D.  Rumelhart 
(714)  452-2230 

Office  of  Naval  Research.  Psychological  Sciences  Division.  1981  Programs.  I 

Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Systems  Automation  through  Artificial  Intelligence 

AFOSR 
D.  Fox 


Tadc-Oriented  Measurement  Technologies 

AFHRL  (Brooks  AFB) 

H.W.  Ruck 
(512)  536-3648 


Air  Force  Human  Resources  Laboratory.  AFHRL  Annual  Report:  FY  81.  Brooks 
AFB,  TX:  AFHRL,  1982. 


70 


Technical  and  Engineering  Services  to  Determine  Automatic  Test  Equipment 
Technological  Use  of  AI  Applications 


NAVAIRENGCEN 
3.  Kunert 
(201)  323-7463 


Theory  of  Graphic  Representation 

NPRDC 
W.E.  Montague 
(714)  225-7121 

Navy  Personnel  Research  and  Development  Center,  NPRDC  FY-82  Programs.  San 
Diego,  CAs  NPRDC,  n.d. 


Topic  and  Theme  Identification  in  Prose  Comprehension 

ONR  (667-423)  University  of  Arizona 

D.  Kieras 
(602)  626-2751 

Office  of  Naval  Research.  Psychological  Sciences  Division,  1981  Programs. 
Arlington,  VA:  Office  of  N aval  Research,  December  19S1. 


Training  Aids  Developed  from  Learning  Guidelines 

TAEG 
R.  Braby 
(305)  646-5673 


Tutoring  and  Problem-Solving  Strategies  in  Intelligent  Computer-Aided 
Instruction 

ONR  (154-436)  Stanford  University 

B.  Buchanan  &  W.  Clancey 
(415) 497-2300 

Office  of  Naval  Research.  Psychological  Sciences  Division,  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Understanding  and  Executing  Instructions 


ONR  (154-461)  Bolt  Beranek  &  Newman,  Inc. 

R.L.  Smith,  3r. 

(617)  492-4841 

Office  of  Naval  Research,  Psychological  Sciences  Division,  1981  Programs. 
Arlington,  VA:  Office  of  Naval  Research,  December  1981. 


Unified  Data  Base  Technology 

AFHRL  (Wright-Patterson  AFB) 

R.N.  Deem 
(513)  255-3771 

Air  Force  Human  Resources  Laboratory.  AFHRL  Annual  Report:  FY  81.  Brooks 
AFB,  TX:  AFHRL,  1982. 


Voice  Technology  as  the  Instructor’s  Assistant 

NTEC  Honeywell 

R.  Ahlers  3.  Brock 

(305)  646-5130  (612)  378-4403 

Defense  Technical  Information  Center.  Work  Unit  Summaries.  Alexandria,  VA: 
DTIC,  1982. 


3ul  81 -Oct  82 
$35,000 


